[==========] Running 7 tests from 2 test suites.
[----------] Global test environment set-up.
[----------] 5 tests from LeaderRebalancerTest
[ RUN ] LeaderRebalancerTest.FunctionalTestForDivided
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20260925 03:02:48.335119 23300 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.22.193.62:42563
I20260925 03:02:48.337348 23300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 03:02:48.338703 23300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 03:02:48.379547 23306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:02:48.379770 23307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 03:02:48.385066 23300 server_base.cc:1034] running on GCE node
W20260925 03:02:48.379782 23309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 03:02:49.637755 23300 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260925 03:02:49.638110 23300 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260925 03:02:49.638315 23300 hybrid_clock.cc:648] HybridClock initialized: now 1790305369638292 us; error 0 us; skew 500 ppm
I20260925 03:02:49.639500 23300 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 03:02:49.650767 23300 webserver.cc:460] Webserver started at http://127.22.193.62:34227/ using document root <none> and password file <none>
I20260925 03:02:49.652037 23300 fs_manager.cc:359] Metadata directory not provided
I20260925 03:02:49.652285 23300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 03:02:49.652848 23300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 03:02:49.661070 23300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklAj8hX/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1790305368086980-23300-0/minicluster-data/master-0-root/instance:
uuid: "d0542f50245c40929690275ab889db50"
format_stamp: "Formatted at 2026-09-25 03:02:49 on dist-test-slave-cswq"
I20260925 03:02:49.671463 23300 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.001s sys 0.007s
I20260925 03:02:49.678858 23315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 03:02:49.680477 23300 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260925 03:02:49.680994 23300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklAj8hX/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1790305368086980-23300-0/minicluster-data/master-0-root
uuid: "d0542f50245c40929690275ab889db50"
format_stamp: "Formatted at 2026-09-25 03:02:49 on dist-test-slave-cswq"
I20260925 03:02:49.681413 23300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklAj8hX/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1790305368086980-23300-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-tasklAj8hX/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1790305368086980-23300-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-tasklAj8hX/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1790305368086980-23300-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 03:02:49.736209 23300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 03:02:49.738276 23300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 03:02:49.738766 23300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 03:02:49.837211 23300 rpc_server.cc:307] RPC server started. Bound to: 127.22.193.62:42563
I20260925 03:02:49.837287 23366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.193.62:42563 every 8 connection(s)
I20260925 03:02:49.848181 23367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:02:49.881309 23367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0542f50245c40929690275ab889db50: Bootstrap starting.
I20260925 03:02:49.889665 23367 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0542f50245c40929690275ab889db50: Neither blocks nor log segments found. Creating new log.
I20260925 03:02:49.892048 23367 log.cc:824] T 00000000000000000000000000000000 P d0542f50245c40929690275ab889db50: Log is configured to *not* fsync() on all Append() calls
I20260925 03:02:49.898523 23367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0542f50245c40929690275ab889db50: No bootstrap required, opened a new log
I20260925 03:02:49.927484 23367 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0542f50245c40929690275ab889db50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0542f50245c40929690275ab889db50" member_type: VOTER }
I20260925 03:02:49.928246 23367 raft_consensus.cc:374] T 00000000000000000000000000000000 P d0542f50245c40929690275ab889db50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:02:49.928534 23367 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0542f50245c40929690275ab889db50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0542f50245c40929690275ab889db50, State: Initialized, Role: FOLLOWER
I20260925 03:02:49.929957 23367 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0542f50245c40929690275ab889db50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0542f50245c40929690275ab889db50" member_type: VOTER }
I20260925 03:02:49.930562 23367 raft_consensus.cc:388] T 00000000000000000000000000000000 P d0542f50245c40929690275ab889db50 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 03:02:49.930893 23367 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0542f50245c40929690275ab889db50 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 03:02:49.931270 23367 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0542f50245c40929690275ab889db50 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:02:49.935808 23367 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0542f50245c40929690275ab889db50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0542f50245c40929690275ab889db50" member_type: VOTER }
I20260925 03:02:49.937039 23367 leader_election.cc:302] T 00000000000000000000000000000000 P d0542f50245c40929690275ab889db50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d0542f50245c40929690275ab889db50; no voters:
I20260925 03:02:49.939677 23367 leader_election.cc:288] T 00000000000000000000000000000000 P d0542f50245c40929690275ab889db50 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 03:02:49.940024 23370 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d0542f50245c40929690275ab889db50 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:02:49.943722 23370 raft_consensus.cc:686] T 00000000000000000000000000000000 P d0542f50245c40929690275ab889db50 [term 1 LEADER]: Becoming Leader. State: Replica: d0542f50245c40929690275ab889db50, State: Running, Role: LEADER
I20260925 03:02:49.945098 23370 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0542f50245c40929690275ab889db50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0542f50245c40929690275ab889db50" member_type: VOTER }
I20260925 03:02:49.946254 23367 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0542f50245c40929690275ab889db50 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 03:02:49.962069 23372 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0542f50245c40929690275ab889db50 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0542f50245c40929690275ab889db50. Latest consensus state: current_term: 1 leader_uuid: "d0542f50245c40929690275ab889db50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0542f50245c40929690275ab889db50" member_type: VOTER } }
I20260925 03:02:49.962127 23371 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0542f50245c40929690275ab889db50 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d0542f50245c40929690275ab889db50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0542f50245c40929690275ab889db50" member_type: VOTER } }
I20260925 03:02:49.963069 23372 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0542f50245c40929690275ab889db50 [sys.catalog]: This master's current role is: LEADER
I20260925 03:02:49.963303 23371 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0542f50245c40929690275ab889db50 [sys.catalog]: This master's current role is: LEADER
I20260925 03:02:49.967633 23378 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 03:02:49.986212 23378 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 03:02:49.989634 23300 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260925 03:02:50.010511 23378 catalog_manager.cc:1347] Generated new cluster ID: 4956db393766469fa018626e9c3f8c54
I20260925 03:02:50.011112 23378 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 03:02:50.037786 23378 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 03:02:50.039794 23378 catalog_manager.cc:1504] Loading token signing keys...
I20260925 03:02:50.057192 23378 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d0542f50245c40929690275ab889db50: Generated new TSK 0
I20260925 03:02:50.058389 23378 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 03:02:50.121459 23300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 03:02:50.132647 23389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:02:50.135217 23390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:02:50.139472 23392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 03:02:50.140130 23300 server_base.cc:1034] running on GCE node
I20260925 03:02:50.141012 23300 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260925 03:02:50.141204 23300 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260925 03:02:50.141363 23300 hybrid_clock.cc:648] HybridClock initialized: now 1790305370141348 us; error 0 us; skew 500 ppm
I20260925 03:02:50.141942 23300 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 03:02:50.148669 23300 webserver.cc:460] Webserver started at http://127.22.193.1:46443/ using document root <none> and password file <none>
I20260925 03:02:50.149260 23300 fs_manager.cc:359] Metadata directory not provided
I20260925 03:02:50.149454 23300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 03:02:50.149727 23300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 03:02:50.151378 23300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklAj8hX/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1790305368086980-23300-0/minicluster-data/ts-0-root/instance:
uuid: "fa551b23283b4034b256ba25acec541b"
format_stamp: "Formatted at 2026-09-25 03:02:50 on dist-test-slave-cswq"
I20260925 03:02:50.158066 23300 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.003s
I20260925 03:02:50.163960 23397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 03:02:50.164997 23300 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.002s
I20260925 03:02:50.165366 23300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklAj8hX/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1790305368086980-23300-0/minicluster-data/ts-0-root
uuid: "fa551b23283b4034b256ba25acec541b"
format_stamp: "Formatted at 2026-09-25 03:02:50 on dist-test-slave-cswq"
I20260925 03:02:50.165772 23300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklAj8hX/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1790305368086980-23300-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-tasklAj8hX/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1790305368086980-23300-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-tasklAj8hX/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1790305368086980-23300-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 03:02:50.200331 23300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 03:02:50.202113 23300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 03:02:50.206766 23300 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 03:02:50.213124 23300 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 03:02:50.213513 23300 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 03:02:50.213836 23300 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 03:02:50.214053 23300 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 03:02:50.368257 23300 rpc_server.cc:307] RPC server started. Bound to: 127.22.193.1:42181
I20260925 03:02:50.368307 23459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.193.1:42181 every 8 connection(s)
I20260925 03:02:50.377616 23300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 03:02:50.390532 23464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:02:50.393985 23465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 03:02:50.405804 23300 server_base.cc:1034] running on GCE node
W20260925 03:02:50.407631 23467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 03:02:50.408782 23300 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260925 03:02:50.409106 23300 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260925 03:02:50.409359 23300 hybrid_clock.cc:648] HybridClock initialized: now 1790305370409340 us; error 0 us; skew 500 ppm
I20260925 03:02:50.410169 23300 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 03:02:50.413908 23300 webserver.cc:460] Webserver started at http://127.22.193.2:38465/ using document root <none> and password file <none>
I20260925 03:02:50.414595 23300 fs_manager.cc:359] Metadata directory not provided
I20260925 03:02:50.414758 23300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 03:02:50.414947 23300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 03:02:50.416112 23300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklAj8hX/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1790305368086980-23300-0/minicluster-data/ts-1-root/instance:
uuid: "2edab657a33a4a0abfbd3262b1668483"
format_stamp: "Formatted at 2026-09-25 03:02:50 on dist-test-slave-cswq"
I20260925 03:02:50.417845 23460 heartbeater.cc:347] Connected to a master server at 127.22.193.62:42563
I20260925 03:02:50.418416 23460 heartbeater.cc:464] Registering TS with master...
I20260925 03:02:50.420014 23460 heartbeater.cc:511] Master 127.22.193.62:42563 requested a full tablet report, sending...
I20260925 03:02:50.422180 23300 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260925 03:02:50.424330 23332 ts_manager.cc:194] Registered new tserver with Master: fa551b23283b4034b256ba25acec541b (127.22.193.1:42181)
I20260925 03:02:50.427888 23472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 03:02:50.428747 23300 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260925 03:02:50.429028 23300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklAj8hX/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1790305368086980-23300-0/minicluster-data/ts-1-root
uuid: "2edab657a33a4a0abfbd3262b1668483"
format_stamp: "Formatted at 2026-09-25 03:02:50 on dist-test-slave-cswq"
I20260925 03:02:50.429296 23300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklAj8hX/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1790305368086980-23300-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-tasklAj8hX/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1790305368086980-23300-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-tasklAj8hX/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1790305368086980-23300-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 03:02:50.429826 23332 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53924
I20260925 03:02:50.462235 23300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 03:02:50.463477 23300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 03:02:50.465242 23300 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 03:02:50.467406 23300 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 03:02:50.467594 23300 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 03:02:50.467809 23300 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 03:02:50.467965 23300 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 03:02:50.581115 23300 rpc_server.cc:307] RPC server started. Bound to: 127.22.193.2:35431
I20260925 03:02:50.581185 23534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.193.2:35431 every 8 connection(s)
I20260925 03:02:50.589040 23300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 03:02:50.600921 23538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:02:50.606163 23539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 03:02:50.608143 23535 heartbeater.cc:347] Connected to a master server at 127.22.193.62:42563
I20260925 03:02:50.608738 23535 heartbeater.cc:464] Registering TS with master...
I20260925 03:02:50.609607 23535 heartbeater.cc:511] Master 127.22.193.62:42563 requested a full tablet report, sending...
I20260925 03:02:50.611129 23300 server_base.cc:1034] running on GCE node
I20260925 03:02:50.611940 23332 ts_manager.cc:194] Registered new tserver with Master: 2edab657a33a4a0abfbd3262b1668483 (127.22.193.2:35431)
W20260925 03:02:50.612596 23542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 03:02:50.613713 23300 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260925 03:02:50.614039 23300 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260925 03:02:50.614065 23332 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53928
I20260925 03:02:50.614410 23300 hybrid_clock.cc:648] HybridClock initialized: now 1790305370614365 us; error 0 us; skew 500 ppm
I20260925 03:02:50.615159 23300 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 03:02:50.618221 23300 webserver.cc:460] Webserver started at http://127.22.193.3:45555/ using document root <none> and password file <none>
I20260925 03:02:50.618832 23300 fs_manager.cc:359] Metadata directory not provided
I20260925 03:02:50.619059 23300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 03:02:50.619381 23300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 03:02:50.620813 23300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklAj8hX/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1790305368086980-23300-0/minicluster-data/ts-2-root/instance:
uuid: "b66c1d3fe3174f64b0936c512ee08f48"
format_stamp: "Formatted at 2026-09-25 03:02:50 on dist-test-slave-cswq"
I20260925 03:02:50.626749 23300 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 03:02:50.631035 23546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 03:02:50.631934 23300 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 03:02:50.632273 23300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklAj8hX/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1790305368086980-23300-0/minicluster-data/ts-2-root
uuid: "b66c1d3fe3174f64b0936c512ee08f48"
format_stamp: "Formatted at 2026-09-25 03:02:50 on dist-test-slave-cswq"
I20260925 03:02:50.632619 23300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklAj8hX/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1790305368086980-23300-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-tasklAj8hX/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1790305368086980-23300-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-tasklAj8hX/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1790305368086980-23300-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 03:02:50.653556 23300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 03:02:50.655354 23300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 03:02:50.659380 23300 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 03:02:50.662273 23300 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 03:02:50.662560 23300 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 03:02:50.663228 23300 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 03:02:50.663463 23300 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 03:02:50.799124 23300 rpc_server.cc:307] RPC server started. Bound to: 127.22.193.3:42621
I20260925 03:02:50.799336 23608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.193.3:42621 every 8 connection(s)
I20260925 03:02:50.823558 23609 heartbeater.cc:347] Connected to a master server at 127.22.193.62:42563
I20260925 03:02:50.823920 23609 heartbeater.cc:464] Registering TS with master...
I20260925 03:02:50.824785 23609 heartbeater.cc:511] Master 127.22.193.62:42563 requested a full tablet report, sending...
I20260925 03:02:50.826609 23332 ts_manager.cc:194] Registered new tserver with Master: b66c1d3fe3174f64b0936c512ee08f48 (127.22.193.3:42621)
I20260925 03:02:50.827018 23300 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.019770662s
I20260925 03:02:50.828636 23332 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53938
I20260925 03:02:50.832521 23300 test_util.cc:268] Using random seed: -206897570
I20260925 03:02:50.901948 23332 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53952:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 03:02:50.908198 23332 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260925 03:02:50.972631 23380 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260925 03:02:51.043198 23574 tablet_service.cc:1467] Processing CreateTablet for tablet ef74cafe53e34e1dace8b02d73693521 (DEFAULT_TABLE table=test-workload [id=aa5e45c3623c411eb4724b11e918500d]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260925 03:02:51.045152 23574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef74cafe53e34e1dace8b02d73693521. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:02:51.045183 23572 tablet_service.cc:1467] Processing CreateTablet for tablet 832e06098a3e499f82d53853e26235cf (DEFAULT_TABLE table=test-workload [id=aa5e45c3623c411eb4724b11e918500d]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260925 03:02:51.046994 23572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 832e06098a3e499f82d53853e26235cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:02:51.050977 23571 tablet_service.cc:1467] Processing CreateTablet for tablet 2baf62cd63564f4ba08b201e40e19d55 (DEFAULT_TABLE table=test-workload [id=aa5e45c3623c411eb4724b11e918500d]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260925 03:02:51.052490 23571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2baf62cd63564f4ba08b201e40e19d55. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:02:51.043205 23573 tablet_service.cc:1467] Processing CreateTablet for tablet d01220d66a8c4d34ac86fc3559fa5165 (DEFAULT_TABLE table=test-workload [id=aa5e45c3623c411eb4724b11e918500d]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260925 03:02:51.060786 23425 tablet_service.cc:1467] Processing CreateTablet for tablet ef74cafe53e34e1dace8b02d73693521 (DEFAULT_TABLE table=test-workload [id=aa5e45c3623c411eb4724b11e918500d]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260925 03:02:51.058080 23570 tablet_service.cc:1467] Processing CreateTablet for tablet c30e9a1092fe4d99994480aab8863cf6 (DEFAULT_TABLE table=test-workload [id=aa5e45c3623c411eb4724b11e918500d]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260925 03:02:51.067536 23573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d01220d66a8c4d34ac86fc3559fa5165. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:02:51.067726 23423 tablet_service.cc:1467] Processing CreateTablet for tablet 832e06098a3e499f82d53853e26235cf (DEFAULT_TABLE table=test-workload [id=aa5e45c3623c411eb4724b11e918500d]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260925 03:02:51.064334 23424 tablet_service.cc:1467] Processing CreateTablet for tablet d01220d66a8c4d34ac86fc3559fa5165 (DEFAULT_TABLE table=test-workload [id=aa5e45c3623c411eb4724b11e918500d]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260925 03:02:51.072352 23425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef74cafe53e34e1dace8b02d73693521. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:02:51.080399 23570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c30e9a1092fe4d99994480aab8863cf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:02:51.080354 23423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 832e06098a3e499f82d53853e26235cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:02:51.068280 23569 tablet_service.cc:1467] Processing CreateTablet for tablet fb663a8b564c494f98f56845f8a05d6c (DEFAULT_TABLE table=test-workload [id=aa5e45c3623c411eb4724b11e918500d]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260925 03:02:51.073231 23500 tablet_service.cc:1467] Processing CreateTablet for tablet ef74cafe53e34e1dace8b02d73693521 (DEFAULT_TABLE table=test-workload [id=aa5e45c3623c411eb4724b11e918500d]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260925 03:02:51.080323 23568 tablet_service.cc:1467] Processing CreateTablet for tablet 43de311d912b475eb01e1f7c9a65412a (DEFAULT_TABLE table=test-workload [id=aa5e45c3623c411eb4724b11e918500d]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260925 03:02:51.097900 23496 tablet_service.cc:1467] Processing CreateTablet for tablet c30e9a1092fe4d99994480aab8863cf6 (DEFAULT_TABLE table=test-workload [id=aa5e45c3623c411eb4724b11e918500d]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260925 03:02:51.101891 23496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c30e9a1092fe4d99994480aab8863cf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:02:51.095960 23424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d01220d66a8c4d34ac86fc3559fa5165. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:02:51.085275 23499 tablet_service.cc:1467] Processing CreateTablet for tablet d01220d66a8c4d34ac86fc3559fa5165 (DEFAULT_TABLE table=test-workload [id=aa5e45c3623c411eb4724b11e918500d]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260925 03:02:51.089200 23422 tablet_service.cc:1467] Processing CreateTablet for tablet 2baf62cd63564f4ba08b201e40e19d55 (DEFAULT_TABLE table=test-workload [id=aa5e45c3623c411eb4724b11e918500d]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260925 03:02:51.110020 23567 tablet_service.cc:1467] Processing CreateTablet for tablet 138dfa413a144a0e8911a2b402857cb4 (DEFAULT_TABLE table=test-workload [id=aa5e45c3623c411eb4724b11e918500d]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260925 03:02:51.105484 23569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb663a8b564c494f98f56845f8a05d6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:02:51.113277 23419 tablet_service.cc:1467] Processing CreateTablet for tablet 43de311d912b475eb01e1f7c9a65412a (DEFAULT_TABLE table=test-workload [id=aa5e45c3623c411eb4724b11e918500d]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260925 03:02:51.117736 23420 tablet_service.cc:1467] Processing CreateTablet for tablet fb663a8b564c494f98f56845f8a05d6c (DEFAULT_TABLE table=test-workload [id=aa5e45c3623c411eb4724b11e918500d]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260925 03:02:51.089306 23497 tablet_service.cc:1467] Processing CreateTablet for tablet 2baf62cd63564f4ba08b201e40e19d55 (DEFAULT_TABLE table=test-workload [id=aa5e45c3623c411eb4724b11e918500d]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260925 03:02:51.119021 23421 tablet_service.cc:1467] Processing CreateTablet for tablet c30e9a1092fe4d99994480aab8863cf6 (DEFAULT_TABLE table=test-workload [id=aa5e45c3623c411eb4724b11e918500d]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260925 03:02:51.121064 23566 tablet_service.cc:1467] Processing CreateTablet for tablet 9de619025d19432c8df31a19bb6fc446 (DEFAULT_TABLE table=test-workload [id=aa5e45c3623c411eb4724b11e918500d]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260925 03:02:51.122920 23498 tablet_service.cc:1467] Processing CreateTablet for tablet 832e06098a3e499f82d53853e26235cf (DEFAULT_TABLE table=test-workload [id=aa5e45c3623c411eb4724b11e918500d]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260925 03:02:51.128654 23500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef74cafe53e34e1dace8b02d73693521. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:02:51.138445 23418 tablet_service.cc:1467] Processing CreateTablet for tablet 138dfa413a144a0e8911a2b402857cb4 (DEFAULT_TABLE table=test-workload [id=aa5e45c3623c411eb4724b11e918500d]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260925 03:02:51.136049 23566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9de619025d19432c8df31a19bb6fc446. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:02:51.143249 23568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43de311d912b475eb01e1f7c9a65412a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:02:51.162513 23418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 138dfa413a144a0e8911a2b402857cb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:02:51.173218 23567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 138dfa413a144a0e8911a2b402857cb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:02:51.172541 23497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2baf62cd63564f4ba08b201e40e19d55. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:02:51.178143 23417 tablet_service.cc:1467] Processing CreateTablet for tablet 9de619025d19432c8df31a19bb6fc446 (DEFAULT_TABLE table=test-workload [id=aa5e45c3623c411eb4724b11e918500d]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260925 03:02:51.183804 23495 tablet_service.cc:1467] Processing CreateTablet for tablet fb663a8b564c494f98f56845f8a05d6c (DEFAULT_TABLE table=test-workload [id=aa5e45c3623c411eb4724b11e918500d]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260925 03:02:51.190053 23499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d01220d66a8c4d34ac86fc3559fa5165. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:02:51.187471 23419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43de311d912b475eb01e1f7c9a65412a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:02:51.201464 23493 tablet_service.cc:1467] Processing CreateTablet for tablet 138dfa413a144a0e8911a2b402857cb4 (DEFAULT_TABLE table=test-workload [id=aa5e45c3623c411eb4724b11e918500d]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260925 03:02:51.194121 23494 tablet_service.cc:1467] Processing CreateTablet for tablet 43de311d912b475eb01e1f7c9a65412a (DEFAULT_TABLE table=test-workload [id=aa5e45c3623c411eb4724b11e918500d]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260925 03:02:51.202793 23495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb663a8b564c494f98f56845f8a05d6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:02:51.211232 23492 tablet_service.cc:1467] Processing CreateTablet for tablet 9de619025d19432c8df31a19bb6fc446 (DEFAULT_TABLE table=test-workload [id=aa5e45c3623c411eb4724b11e918500d]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260925 03:02:51.213552 23493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 138dfa413a144a0e8911a2b402857cb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:02:51.214696 23494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43de311d912b475eb01e1f7c9a65412a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:02:51.211742 23422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2baf62cd63564f4ba08b201e40e19d55. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:02:51.217672 23492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9de619025d19432c8df31a19bb6fc446. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:02:51.236930 23417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9de619025d19432c8df31a19bb6fc446. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:02:51.244159 23498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 832e06098a3e499f82d53853e26235cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:02:51.244946 23421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c30e9a1092fe4d99994480aab8863cf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:02:51.283788 23420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb663a8b564c494f98f56845f8a05d6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:02:51.371922 23627 tablet_bootstrap.cc:492] T 9de619025d19432c8df31a19bb6fc446 P b66c1d3fe3174f64b0936c512ee08f48: Bootstrap starting.
I20260925 03:02:51.387705 23629 tablet_bootstrap.cc:492] T ef74cafe53e34e1dace8b02d73693521 P fa551b23283b4034b256ba25acec541b: Bootstrap starting.
I20260925 03:02:51.387048 23628 tablet_bootstrap.cc:492] T 43de311d912b475eb01e1f7c9a65412a P 2edab657a33a4a0abfbd3262b1668483: Bootstrap starting.
I20260925 03:02:51.402796 23629 tablet_bootstrap.cc:654] T ef74cafe53e34e1dace8b02d73693521 P fa551b23283b4034b256ba25acec541b: Neither blocks nor log segments found. Creating new log.
I20260925 03:02:51.435701 23460 heartbeater.cc:503] Master 127.22.193.62:42563 was elected leader, sending a full tablet report...
I20260925 03:02:51.490077 23629 tablet_bootstrap.cc:492] T ef74cafe53e34e1dace8b02d73693521 P fa551b23283b4034b256ba25acec541b: No bootstrap required, opened a new log
I20260925 03:02:51.490797 23629 ts_tablet_manager.cc:1397] T ef74cafe53e34e1dace8b02d73693521 P fa551b23283b4034b256ba25acec541b: Time spent bootstrapping tablet: real 0.104s user 0.043s sys 0.019s
I20260925 03:02:51.491982 23628 tablet_bootstrap.cc:654] T 43de311d912b475eb01e1f7c9a65412a P 2edab657a33a4a0abfbd3262b1668483: Neither blocks nor log segments found. Creating new log.
I20260925 03:02:51.496196 23627 tablet_bootstrap.cc:654] T 9de619025d19432c8df31a19bb6fc446 P b66c1d3fe3174f64b0936c512ee08f48: Neither blocks nor log segments found. Creating new log.
I20260925 03:02:51.498016 23629 raft_consensus.cc:348] T ef74cafe53e34e1dace8b02d73693521 P fa551b23283b4034b256ba25acec541b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } } peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } }
I20260925 03:02:51.498834 23629 raft_consensus.cc:374] T ef74cafe53e34e1dace8b02d73693521 P fa551b23283b4034b256ba25acec541b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:02:51.499122 23629 raft_consensus.cc:729] T ef74cafe53e34e1dace8b02d73693521 P fa551b23283b4034b256ba25acec541b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa551b23283b4034b256ba25acec541b, State: Initialized, Role: FOLLOWER
I20260925 03:02:51.499771 23629 consensus_queue.cc:260] T ef74cafe53e34e1dace8b02d73693521 P fa551b23283b4034b256ba25acec541b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } } peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } }
I20260925 03:02:51.515681 23627 tablet_bootstrap.cc:492] T 9de619025d19432c8df31a19bb6fc446 P b66c1d3fe3174f64b0936c512ee08f48: No bootstrap required, opened a new log
I20260925 03:02:51.516999 23627 ts_tablet_manager.cc:1397] T 9de619025d19432c8df31a19bb6fc446 P b66c1d3fe3174f64b0936c512ee08f48: Time spent bootstrapping tablet: real 0.160s user 0.000s sys 0.020s
I20260925 03:02:51.519730 23627 raft_consensus.cc:348] T 9de619025d19432c8df31a19bb6fc446 P b66c1d3fe3174f64b0936c512ee08f48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:02:51.520291 23627 raft_consensus.cc:374] T 9de619025d19432c8df31a19bb6fc446 P b66c1d3fe3174f64b0936c512ee08f48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:02:51.520533 23627 raft_consensus.cc:729] T 9de619025d19432c8df31a19bb6fc446 P b66c1d3fe3174f64b0936c512ee08f48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b66c1d3fe3174f64b0936c512ee08f48, State: Initialized, Role: FOLLOWER
I20260925 03:02:51.521200 23627 consensus_queue.cc:260] T 9de619025d19432c8df31a19bb6fc446 P b66c1d3fe3174f64b0936c512ee08f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:02:51.523607 23629 ts_tablet_manager.cc:1428] T ef74cafe53e34e1dace8b02d73693521 P fa551b23283b4034b256ba25acec541b: Time spent starting tablet: real 0.031s user 0.017s sys 0.011s
I20260925 03:02:51.523756 23628 tablet_bootstrap.cc:492] T 43de311d912b475eb01e1f7c9a65412a P 2edab657a33a4a0abfbd3262b1668483: No bootstrap required, opened a new log
I20260925 03:02:51.524307 23628 ts_tablet_manager.cc:1397] T 43de311d912b475eb01e1f7c9a65412a P 2edab657a33a4a0abfbd3262b1668483: Time spent bootstrapping tablet: real 0.161s user 0.008s sys 0.025s
I20260925 03:02:51.524621 23629 tablet_bootstrap.cc:492] T 43de311d912b475eb01e1f7c9a65412a P fa551b23283b4034b256ba25acec541b: Bootstrap starting.
I20260925 03:02:51.527549 23628 raft_consensus.cc:348] T 43de311d912b475eb01e1f7c9a65412a P 2edab657a33a4a0abfbd3262b1668483 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:02:51.528378 23628 raft_consensus.cc:374] T 43de311d912b475eb01e1f7c9a65412a P 2edab657a33a4a0abfbd3262b1668483 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:02:51.528749 23628 raft_consensus.cc:729] T 43de311d912b475eb01e1f7c9a65412a P 2edab657a33a4a0abfbd3262b1668483 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2edab657a33a4a0abfbd3262b1668483, State: Initialized, Role: FOLLOWER
I20260925 03:02:51.529527 23628 consensus_queue.cc:260] T 43de311d912b475eb01e1f7c9a65412a P 2edab657a33a4a0abfbd3262b1668483 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:02:51.531756 23629 tablet_bootstrap.cc:654] T 43de311d912b475eb01e1f7c9a65412a P fa551b23283b4034b256ba25acec541b: Neither blocks nor log segments found. Creating new log.
I20260925 03:02:51.544072 23609 heartbeater.cc:503] Master 127.22.193.62:42563 was elected leader, sending a full tablet report...
I20260925 03:02:51.545583 23627 ts_tablet_manager.cc:1428] T 9de619025d19432c8df31a19bb6fc446 P b66c1d3fe3174f64b0936c512ee08f48: Time spent starting tablet: real 0.028s user 0.004s sys 0.008s
I20260925 03:02:51.547644 23627 tablet_bootstrap.cc:492] T 2baf62cd63564f4ba08b201e40e19d55 P b66c1d3fe3174f64b0936c512ee08f48: Bootstrap starting.
I20260925 03:02:51.548364 23628 ts_tablet_manager.cc:1428] T 43de311d912b475eb01e1f7c9a65412a P 2edab657a33a4a0abfbd3262b1668483: Time spent starting tablet: real 0.024s user 0.014s sys 0.008s
I20260925 03:02:51.549221 23535 heartbeater.cc:503] Master 127.22.193.62:42563 was elected leader, sending a full tablet report...
I20260925 03:02:51.549440 23628 tablet_bootstrap.cc:492] T ef74cafe53e34e1dace8b02d73693521 P 2edab657a33a4a0abfbd3262b1668483: Bootstrap starting.
I20260925 03:02:51.557403 23629 tablet_bootstrap.cc:492] T 43de311d912b475eb01e1f7c9a65412a P fa551b23283b4034b256ba25acec541b: No bootstrap required, opened a new log
I20260925 03:02:51.557996 23629 ts_tablet_manager.cc:1397] T 43de311d912b475eb01e1f7c9a65412a P fa551b23283b4034b256ba25acec541b: Time spent bootstrapping tablet: real 0.034s user 0.007s sys 0.011s
W20260925 03:02:51.560253 23610 tablet.cc:2369] T 9de619025d19432c8df31a19bb6fc446 P b66c1d3fe3174f64b0936c512ee08f48: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 03:02:51.562683 23629 raft_consensus.cc:348] T 43de311d912b475eb01e1f7c9a65412a P fa551b23283b4034b256ba25acec541b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:02:51.563459 23629 raft_consensus.cc:374] T 43de311d912b475eb01e1f7c9a65412a P fa551b23283b4034b256ba25acec541b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:02:51.563786 23629 raft_consensus.cc:729] T 43de311d912b475eb01e1f7c9a65412a P fa551b23283b4034b256ba25acec541b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa551b23283b4034b256ba25acec541b, State: Initialized, Role: FOLLOWER
I20260925 03:02:51.564483 23629 consensus_queue.cc:260] T 43de311d912b475eb01e1f7c9a65412a P fa551b23283b4034b256ba25acec541b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:02:51.569482 23627 tablet_bootstrap.cc:654] T 2baf62cd63564f4ba08b201e40e19d55 P b66c1d3fe3174f64b0936c512ee08f48: Neither blocks nor log segments found. Creating new log.
I20260925 03:02:51.566164 23635 raft_consensus.cc:482] T 43de311d912b475eb01e1f7c9a65412a P 2edab657a33a4a0abfbd3262b1668483 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:02:51.572844 23635 raft_consensus.cc:504] T 43de311d912b475eb01e1f7c9a65412a P 2edab657a33a4a0abfbd3262b1668483 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:02:51.576001 23629 ts_tablet_manager.cc:1428] T 43de311d912b475eb01e1f7c9a65412a P fa551b23283b4034b256ba25acec541b: Time spent starting tablet: real 0.018s user 0.011s sys 0.004s
I20260925 03:02:51.572279 23628 tablet_bootstrap.cc:654] T ef74cafe53e34e1dace8b02d73693521 P 2edab657a33a4a0abfbd3262b1668483: Neither blocks nor log segments found. Creating new log.
I20260925 03:02:51.578471 23635 leader_election.cc:288] T 43de311d912b475eb01e1f7c9a65412a P 2edab657a33a4a0abfbd3262b1668483 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa551b23283b4034b256ba25acec541b (127.22.193.1:42181), b66c1d3fe3174f64b0936c512ee08f48 (127.22.193.3:42621)
I20260925 03:02:51.578078 23629 tablet_bootstrap.cc:492] T 832e06098a3e499f82d53853e26235cf P fa551b23283b4034b256ba25acec541b: Bootstrap starting.
I20260925 03:02:51.601713 23627 tablet_bootstrap.cc:492] T 2baf62cd63564f4ba08b201e40e19d55 P b66c1d3fe3174f64b0936c512ee08f48: No bootstrap required, opened a new log
I20260925 03:02:51.604557 23627 ts_tablet_manager.cc:1397] T 2baf62cd63564f4ba08b201e40e19d55 P b66c1d3fe3174f64b0936c512ee08f48: Time spent bootstrapping tablet: real 0.057s user 0.015s sys 0.003s
I20260925 03:02:51.609211 23629 tablet_bootstrap.cc:654] T 832e06098a3e499f82d53853e26235cf P fa551b23283b4034b256ba25acec541b: Neither blocks nor log segments found. Creating new log.
I20260925 03:02:51.613260 23627 raft_consensus.cc:348] T 2baf62cd63564f4ba08b201e40e19d55 P b66c1d3fe3174f64b0936c512ee08f48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:02:51.614015 23627 raft_consensus.cc:374] T 2baf62cd63564f4ba08b201e40e19d55 P b66c1d3fe3174f64b0936c512ee08f48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:02:51.614322 23627 raft_consensus.cc:729] T 2baf62cd63564f4ba08b201e40e19d55 P b66c1d3fe3174f64b0936c512ee08f48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b66c1d3fe3174f64b0936c512ee08f48, State: Initialized, Role: FOLLOWER
I20260925 03:02:51.615608 23627 consensus_queue.cc:260] T 2baf62cd63564f4ba08b201e40e19d55 P b66c1d3fe3174f64b0936c512ee08f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:02:51.619843 23628 tablet_bootstrap.cc:492] T ef74cafe53e34e1dace8b02d73693521 P 2edab657a33a4a0abfbd3262b1668483: No bootstrap required, opened a new log
I20260925 03:02:51.621367 23628 ts_tablet_manager.cc:1397] T ef74cafe53e34e1dace8b02d73693521 P 2edab657a33a4a0abfbd3262b1668483: Time spent bootstrapping tablet: real 0.072s user 0.013s sys 0.008s
I20260925 03:02:51.623211 23627 ts_tablet_manager.cc:1428] T 2baf62cd63564f4ba08b201e40e19d55 P b66c1d3fe3174f64b0936c512ee08f48: Time spent starting tablet: real 0.018s user 0.009s sys 0.000s
I20260925 03:02:51.624167 23627 tablet_bootstrap.cc:492] T d01220d66a8c4d34ac86fc3559fa5165 P b66c1d3fe3174f64b0936c512ee08f48: Bootstrap starting.
I20260925 03:02:51.624643 23628 raft_consensus.cc:348] T ef74cafe53e34e1dace8b02d73693521 P 2edab657a33a4a0abfbd3262b1668483 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } } peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } }
I20260925 03:02:51.625432 23628 raft_consensus.cc:374] T ef74cafe53e34e1dace8b02d73693521 P 2edab657a33a4a0abfbd3262b1668483 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:02:51.625713 23628 raft_consensus.cc:729] T ef74cafe53e34e1dace8b02d73693521 P 2edab657a33a4a0abfbd3262b1668483 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2edab657a33a4a0abfbd3262b1668483, State: Initialized, Role: FOLLOWER
I20260925 03:02:51.626418 23634 raft_consensus.cc:482] T 9de619025d19432c8df31a19bb6fc446 P b66c1d3fe3174f64b0936c512ee08f48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:02:51.627029 23634 raft_consensus.cc:504] T 9de619025d19432c8df31a19bb6fc446 P b66c1d3fe3174f64b0936c512ee08f48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:02:51.627107 23628 consensus_queue.cc:260] T ef74cafe53e34e1dace8b02d73693521 P 2edab657a33a4a0abfbd3262b1668483 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } } peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } }
I20260925 03:02:51.633438 23634 leader_election.cc:288] T 9de619025d19432c8df31a19bb6fc446 P b66c1d3fe3174f64b0936c512ee08f48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa551b23283b4034b256ba25acec541b (127.22.193.1:42181), 2edab657a33a4a0abfbd3262b1668483 (127.22.193.2:35431)
I20260925 03:02:51.633981 23435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43de311d912b475eb01e1f7c9a65412a" candidate_uuid: "2edab657a33a4a0abfbd3262b1668483" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa551b23283b4034b256ba25acec541b" is_pre_election: true
I20260925 03:02:51.636408 23628 ts_tablet_manager.cc:1428] T ef74cafe53e34e1dace8b02d73693521 P 2edab657a33a4a0abfbd3262b1668483: Time spent starting tablet: real 0.015s user 0.006s sys 0.002s
I20260925 03:02:51.637449 23435 raft_consensus.cc:2454] T 43de311d912b475eb01e1f7c9a65412a P fa551b23283b4034b256ba25acec541b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2edab657a33a4a0abfbd3262b1668483 in term 0.
I20260925 03:02:51.641345 23628 tablet_bootstrap.cc:492] T d01220d66a8c4d34ac86fc3559fa5165 P 2edab657a33a4a0abfbd3262b1668483: Bootstrap starting.
I20260925 03:02:51.663913 23473 leader_election.cc:302] T 43de311d912b475eb01e1f7c9a65412a P 2edab657a33a4a0abfbd3262b1668483 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2edab657a33a4a0abfbd3262b1668483, fa551b23283b4034b256ba25acec541b; no voters:
I20260925 03:02:51.665812 23637 raft_consensus.cc:2781] T 43de311d912b475eb01e1f7c9a65412a P 2edab657a33a4a0abfbd3262b1668483 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 03:02:51.666235 23637 raft_consensus.cc:482] T 43de311d912b475eb01e1f7c9a65412a P 2edab657a33a4a0abfbd3262b1668483 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:02:51.666572 23637 raft_consensus.cc:3037] T 43de311d912b475eb01e1f7c9a65412a P 2edab657a33a4a0abfbd3262b1668483 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:02:51.678110 23628 tablet_bootstrap.cc:654] T d01220d66a8c4d34ac86fc3559fa5165 P 2edab657a33a4a0abfbd3262b1668483: Neither blocks nor log segments found. Creating new log.
I20260925 03:02:51.691947 23584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43de311d912b475eb01e1f7c9a65412a" candidate_uuid: "2edab657a33a4a0abfbd3262b1668483" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b66c1d3fe3174f64b0936c512ee08f48" is_pre_election: true
W20260925 03:02:51.695147 23473 leader_election.cc:341] T 43de311d912b475eb01e1f7c9a65412a P 2edab657a33a4a0abfbd3262b1668483 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b66c1d3fe3174f64b0936c512ee08f48 (127.22.193.3:42621): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:02:51.698567 23627 tablet_bootstrap.cc:654] T d01220d66a8c4d34ac86fc3559fa5165 P b66c1d3fe3174f64b0936c512ee08f48: Neither blocks nor log segments found. Creating new log.
I20260925 03:02:51.699544 23637 raft_consensus.cc:504] T 43de311d912b475eb01e1f7c9a65412a P 2edab657a33a4a0abfbd3262b1668483 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:02:51.709065 23629 tablet_bootstrap.cc:492] T 832e06098a3e499f82d53853e26235cf P fa551b23283b4034b256ba25acec541b: No bootstrap required, opened a new log
I20260925 03:02:51.710996 23629 ts_tablet_manager.cc:1397] T 832e06098a3e499f82d53853e26235cf P fa551b23283b4034b256ba25acec541b: Time spent bootstrapping tablet: real 0.134s user 0.015s sys 0.005s
I20260925 03:02:51.712291 23435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43de311d912b475eb01e1f7c9a65412a" candidate_uuid: "2edab657a33a4a0abfbd3262b1668483" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa551b23283b4034b256ba25acec541b"
I20260925 03:02:51.712783 23637 leader_election.cc:288] T 43de311d912b475eb01e1f7c9a65412a P 2edab657a33a4a0abfbd3262b1668483 [CANDIDATE]: Term 1 election: Requested vote from peers fa551b23283b4034b256ba25acec541b (127.22.193.1:42181), b66c1d3fe3174f64b0936c512ee08f48 (127.22.193.3:42621)
I20260925 03:02:51.714277 23435 raft_consensus.cc:3037] T 43de311d912b475eb01e1f7c9a65412a P fa551b23283b4034b256ba25acec541b [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:02:51.715925 23584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43de311d912b475eb01e1f7c9a65412a" candidate_uuid: "2edab657a33a4a0abfbd3262b1668483" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b66c1d3fe3174f64b0936c512ee08f48"
I20260925 03:02:51.718467 23628 tablet_bootstrap.cc:492] T d01220d66a8c4d34ac86fc3559fa5165 P 2edab657a33a4a0abfbd3262b1668483: No bootstrap required, opened a new log
W20260925 03:02:51.718561 23473 leader_election.cc:341] T 43de311d912b475eb01e1f7c9a65412a P 2edab657a33a4a0abfbd3262b1668483 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b66c1d3fe3174f64b0936c512ee08f48 (127.22.193.3:42621): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:02:51.719084 23628 ts_tablet_manager.cc:1397] T d01220d66a8c4d34ac86fc3559fa5165 P 2edab657a33a4a0abfbd3262b1668483: Time spent bootstrapping tablet: real 0.079s user 0.011s sys 0.008s
I20260925 03:02:51.721804 23629 raft_consensus.cc:348] T 832e06098a3e499f82d53853e26235cf P fa551b23283b4034b256ba25acec541b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:02:51.723739 23629 raft_consensus.cc:374] T 832e06098a3e499f82d53853e26235cf P fa551b23283b4034b256ba25acec541b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:02:51.724471 23629 raft_consensus.cc:729] T 832e06098a3e499f82d53853e26235cf P fa551b23283b4034b256ba25acec541b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa551b23283b4034b256ba25acec541b, State: Initialized, Role: FOLLOWER
I20260925 03:02:51.727368 23628 raft_consensus.cc:348] T d01220d66a8c4d34ac86fc3559fa5165 P 2edab657a33a4a0abfbd3262b1668483 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:02:51.728845 23628 raft_consensus.cc:374] T d01220d66a8c4d34ac86fc3559fa5165 P 2edab657a33a4a0abfbd3262b1668483 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:02:51.727679 23629 consensus_queue.cc:260] T 832e06098a3e499f82d53853e26235cf P fa551b23283b4034b256ba25acec541b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:02:51.729120 23628 raft_consensus.cc:729] T d01220d66a8c4d34ac86fc3559fa5165 P 2edab657a33a4a0abfbd3262b1668483 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2edab657a33a4a0abfbd3262b1668483, State: Initialized, Role: FOLLOWER
I20260925 03:02:51.731765 23435 raft_consensus.cc:2454] T 43de311d912b475eb01e1f7c9a65412a P fa551b23283b4034b256ba25acec541b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2edab657a33a4a0abfbd3262b1668483 in term 1.
I20260925 03:02:51.732981 23628 consensus_queue.cc:260] T d01220d66a8c4d34ac86fc3559fa5165 P 2edab657a33a4a0abfbd3262b1668483 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:02:51.735949 23628 ts_tablet_manager.cc:1428] T d01220d66a8c4d34ac86fc3559fa5165 P 2edab657a33a4a0abfbd3262b1668483: Time spent starting tablet: real 0.017s user 0.008s sys 0.000s
I20260925 03:02:51.735916 23473 leader_election.cc:302] T 43de311d912b475eb01e1f7c9a65412a P 2edab657a33a4a0abfbd3262b1668483 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2edab657a33a4a0abfbd3262b1668483, fa551b23283b4034b256ba25acec541b; no voters: b66c1d3fe3174f64b0936c512ee08f48
I20260925 03:02:51.737036 23628 tablet_bootstrap.cc:492] T 2baf62cd63564f4ba08b201e40e19d55 P 2edab657a33a4a0abfbd3262b1668483: Bootstrap starting.
I20260925 03:02:51.737684 23637 raft_consensus.cc:2781] T 43de311d912b475eb01e1f7c9a65412a P 2edab657a33a4a0abfbd3262b1668483 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:02:51.739328 23637 raft_consensus.cc:686] T 43de311d912b475eb01e1f7c9a65412a P 2edab657a33a4a0abfbd3262b1668483 [term 1 LEADER]: Becoming Leader. State: Replica: 2edab657a33a4a0abfbd3262b1668483, State: Running, Role: LEADER
I20260925 03:02:51.740355 23637 consensus_queue.cc:237] T 43de311d912b475eb01e1f7c9a65412a P 2edab657a33a4a0abfbd3262b1668483 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:02:51.746750 23510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9de619025d19432c8df31a19bb6fc446" candidate_uuid: "b66c1d3fe3174f64b0936c512ee08f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2edab657a33a4a0abfbd3262b1668483" is_pre_election: true
W20260925 03:02:51.751785 23548 leader_election.cc:341] T 9de619025d19432c8df31a19bb6fc446 P b66c1d3fe3174f64b0936c512ee08f48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2edab657a33a4a0abfbd3262b1668483 (127.22.193.2:35431): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:02:51.754081 23629 ts_tablet_manager.cc:1428] T 832e06098a3e499f82d53853e26235cf P fa551b23283b4034b256ba25acec541b: Time spent starting tablet: real 0.042s user 0.010s sys 0.000s
I20260925 03:02:51.757709 23435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9de619025d19432c8df31a19bb6fc446" candidate_uuid: "b66c1d3fe3174f64b0936c512ee08f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa551b23283b4034b256ba25acec541b" is_pre_election: true
I20260925 03:02:51.759431 23629 tablet_bootstrap.cc:492] T 2baf62cd63564f4ba08b201e40e19d55 P fa551b23283b4034b256ba25acec541b: Bootstrap starting.
W20260925 03:02:51.765412 23547 leader_election.cc:341] T 9de619025d19432c8df31a19bb6fc446 P b66c1d3fe3174f64b0936c512ee08f48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fa551b23283b4034b256ba25acec541b (127.22.193.1:42181): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:02:51.766214 23547 leader_election.cc:302] T 9de619025d19432c8df31a19bb6fc446 P b66c1d3fe3174f64b0936c512ee08f48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b66c1d3fe3174f64b0936c512ee08f48; no voters: 2edab657a33a4a0abfbd3262b1668483, fa551b23283b4034b256ba25acec541b
I20260925 03:02:51.768217 23634 raft_consensus.cc:2726] T 9de619025d19432c8df31a19bb6fc446 P b66c1d3fe3174f64b0936c512ee08f48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 03:02:51.773509 23627 tablet_bootstrap.cc:492] T d01220d66a8c4d34ac86fc3559fa5165 P b66c1d3fe3174f64b0936c512ee08f48: No bootstrap required, opened a new log
I20260925 03:02:51.774045 23627 ts_tablet_manager.cc:1397] T d01220d66a8c4d34ac86fc3559fa5165 P b66c1d3fe3174f64b0936c512ee08f48: Time spent bootstrapping tablet: real 0.150s user 0.016s sys 0.007s
I20260925 03:02:51.775182 23629 tablet_bootstrap.cc:654] T 2baf62cd63564f4ba08b201e40e19d55 P fa551b23283b4034b256ba25acec541b: Neither blocks nor log segments found. Creating new log.
I20260925 03:02:51.777205 23627 raft_consensus.cc:348] T d01220d66a8c4d34ac86fc3559fa5165 P b66c1d3fe3174f64b0936c512ee08f48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:02:51.778198 23627 raft_consensus.cc:374] T d01220d66a8c4d34ac86fc3559fa5165 P b66c1d3fe3174f64b0936c512ee08f48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:02:51.778678 23627 raft_consensus.cc:729] T d01220d66a8c4d34ac86fc3559fa5165 P b66c1d3fe3174f64b0936c512ee08f48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b66c1d3fe3174f64b0936c512ee08f48, State: Initialized, Role: FOLLOWER
I20260925 03:02:51.780100 23627 consensus_queue.cc:260] T d01220d66a8c4d34ac86fc3559fa5165 P b66c1d3fe3174f64b0936c512ee08f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:02:51.784421 23628 tablet_bootstrap.cc:654] T 2baf62cd63564f4ba08b201e40e19d55 P 2edab657a33a4a0abfbd3262b1668483: Neither blocks nor log segments found. Creating new log.
I20260925 03:02:51.784937 23627 ts_tablet_manager.cc:1428] T d01220d66a8c4d34ac86fc3559fa5165 P b66c1d3fe3174f64b0936c512ee08f48: Time spent starting tablet: real 0.011s user 0.003s sys 0.006s
I20260925 03:02:51.787412 23627 tablet_bootstrap.cc:492] T ef74cafe53e34e1dace8b02d73693521 P b66c1d3fe3174f64b0936c512ee08f48: Bootstrap starting.
I20260925 03:02:51.798039 23627 tablet_bootstrap.cc:654] T ef74cafe53e34e1dace8b02d73693521 P b66c1d3fe3174f64b0936c512ee08f48: Neither blocks nor log segments found. Creating new log.
I20260925 03:02:51.813731 23332 catalog_manager.cc:5495] T 43de311d912b475eb01e1f7c9a65412a P 2edab657a33a4a0abfbd3262b1668483 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2edab657a33a4a0abfbd3262b1668483 (127.22.193.2). New cstate: current_term: 1 leader_uuid: "2edab657a33a4a0abfbd3262b1668483" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } health_report { overall_health: UNKNOWN } } }
I20260925 03:02:51.819258 23628 tablet_bootstrap.cc:492] T 2baf62cd63564f4ba08b201e40e19d55 P 2edab657a33a4a0abfbd3262b1668483: No bootstrap required, opened a new log
I20260925 03:02:51.819839 23628 ts_tablet_manager.cc:1397] T 2baf62cd63564f4ba08b201e40e19d55 P 2edab657a33a4a0abfbd3262b1668483: Time spent bootstrapping tablet: real 0.083s user 0.014s sys 0.009s
I20260925 03:02:51.822399 23629 tablet_bootstrap.cc:492] T 2baf62cd63564f4ba08b201e40e19d55 P fa551b23283b4034b256ba25acec541b: No bootstrap required, opened a new log
I20260925 03:02:51.822896 23629 ts_tablet_manager.cc:1397] T 2baf62cd63564f4ba08b201e40e19d55 P fa551b23283b4034b256ba25acec541b: Time spent bootstrapping tablet: real 0.065s user 0.014s sys 0.013s
I20260925 03:02:51.822772 23628 raft_consensus.cc:348] T 2baf62cd63564f4ba08b201e40e19d55 P 2edab657a33a4a0abfbd3262b1668483 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:02:51.824055 23628 raft_consensus.cc:374] T 2baf62cd63564f4ba08b201e40e19d55 P 2edab657a33a4a0abfbd3262b1668483 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:02:51.824434 23628 raft_consensus.cc:729] T 2baf62cd63564f4ba08b201e40e19d55 P 2edab657a33a4a0abfbd3262b1668483 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2edab657a33a4a0abfbd3262b1668483, State: Initialized, Role: FOLLOWER
I20260925 03:02:51.825271 23628 consensus_queue.cc:260] T 2baf62cd63564f4ba08b201e40e19d55 P 2edab657a33a4a0abfbd3262b1668483 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:02:51.827028 23629 raft_consensus.cc:348] T 2baf62cd63564f4ba08b201e40e19d55 P fa551b23283b4034b256ba25acec541b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:02:51.827962 23629 raft_consensus.cc:374] T 2baf62cd63564f4ba08b201e40e19d55 P fa551b23283b4034b256ba25acec541b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:02:51.828372 23628 ts_tablet_manager.cc:1428] T 2baf62cd63564f4ba08b201e40e19d55 P 2edab657a33a4a0abfbd3262b1668483: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260925 03:02:51.828366 23629 raft_consensus.cc:729] T 2baf62cd63564f4ba08b201e40e19d55 P fa551b23283b4034b256ba25acec541b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa551b23283b4034b256ba25acec541b, State: Initialized, Role: FOLLOWER
I20260925 03:02:51.829432 23629 consensus_queue.cc:260] T 2baf62cd63564f4ba08b201e40e19d55 P fa551b23283b4034b256ba25acec541b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:02:51.831041 23627 tablet_bootstrap.cc:492] T ef74cafe53e34e1dace8b02d73693521 P b66c1d3fe3174f64b0936c512ee08f48: No bootstrap required, opened a new log
I20260925 03:02:51.831481 23627 ts_tablet_manager.cc:1397] T ef74cafe53e34e1dace8b02d73693521 P b66c1d3fe3174f64b0936c512ee08f48: Time spent bootstrapping tablet: real 0.044s user 0.011s sys 0.011s
I20260925 03:02:51.832420 23628 tablet_bootstrap.cc:492] T 9de619025d19432c8df31a19bb6fc446 P 2edab657a33a4a0abfbd3262b1668483: Bootstrap starting.
I20260925 03:02:51.839143 23629 ts_tablet_manager.cc:1428] T 2baf62cd63564f4ba08b201e40e19d55 P fa551b23283b4034b256ba25acec541b: Time spent starting tablet: real 0.016s user 0.006s sys 0.000s
I20260925 03:02:51.840225 23629 tablet_bootstrap.cc:492] T d01220d66a8c4d34ac86fc3559fa5165 P fa551b23283b4034b256ba25acec541b: Bootstrap starting.
I20260925 03:02:51.845482 23637 raft_consensus.cc:482] T ef74cafe53e34e1dace8b02d73693521 P 2edab657a33a4a0abfbd3262b1668483 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:02:51.845973 23637 raft_consensus.cc:504] T ef74cafe53e34e1dace8b02d73693521 P 2edab657a33a4a0abfbd3262b1668483 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } } peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } }
I20260925 03:02:51.841791 23627 raft_consensus.cc:348] T ef74cafe53e34e1dace8b02d73693521 P b66c1d3fe3174f64b0936c512ee08f48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } } peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } }
I20260925 03:02:51.847088 23627 raft_consensus.cc:374] T ef74cafe53e34e1dace8b02d73693521 P b66c1d3fe3174f64b0936c512ee08f48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:02:51.847395 23627 raft_consensus.cc:729] T ef74cafe53e34e1dace8b02d73693521 P b66c1d3fe3174f64b0936c512ee08f48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b66c1d3fe3174f64b0936c512ee08f48, State: Initialized, Role: FOLLOWER
I20260925 03:02:51.848237 23637 leader_election.cc:288] T ef74cafe53e34e1dace8b02d73693521 P 2edab657a33a4a0abfbd3262b1668483 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b66c1d3fe3174f64b0936c512ee08f48 (127.22.193.3:42621), fa551b23283b4034b256ba25acec541b (127.22.193.1:42181)
I20260925 03:02:51.848186 23627 consensus_queue.cc:260] T ef74cafe53e34e1dace8b02d73693521 P b66c1d3fe3174f64b0936c512ee08f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } } peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } }
I20260925 03:02:51.850176 23584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef74cafe53e34e1dace8b02d73693521" candidate_uuid: "2edab657a33a4a0abfbd3262b1668483" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b66c1d3fe3174f64b0936c512ee08f48" is_pre_election: true
I20260925 03:02:51.850602 23435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef74cafe53e34e1dace8b02d73693521" candidate_uuid: "2edab657a33a4a0abfbd3262b1668483" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa551b23283b4034b256ba25acec541b" is_pre_election: true
I20260925 03:02:51.850814 23584 raft_consensus.cc:2454] T ef74cafe53e34e1dace8b02d73693521 P b66c1d3fe3174f64b0936c512ee08f48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2edab657a33a4a0abfbd3262b1668483 in term 0.
I20260925 03:02:51.851660 23435 raft_consensus.cc:2454] T ef74cafe53e34e1dace8b02d73693521 P fa551b23283b4034b256ba25acec541b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2edab657a33a4a0abfbd3262b1668483 in term 0.
I20260925 03:02:51.852900 23632 raft_consensus.cc:482] T ef74cafe53e34e1dace8b02d73693521 P fa551b23283b4034b256ba25acec541b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:02:51.854065 23627 ts_tablet_manager.cc:1428] T ef74cafe53e34e1dace8b02d73693521 P b66c1d3fe3174f64b0936c512ee08f48: Time spent starting tablet: real 0.022s user 0.007s sys 0.003s
I20260925 03:02:51.853641 23632 raft_consensus.cc:504] T ef74cafe53e34e1dace8b02d73693521 P fa551b23283b4034b256ba25acec541b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } } peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } }
I20260925 03:02:51.855006 23627 tablet_bootstrap.cc:492] T fb663a8b564c494f98f56845f8a05d6c P b66c1d3fe3174f64b0936c512ee08f48: Bootstrap starting.
I20260925 03:02:51.852834 23473 leader_election.cc:302] T ef74cafe53e34e1dace8b02d73693521 P 2edab657a33a4a0abfbd3262b1668483 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2edab657a33a4a0abfbd3262b1668483, b66c1d3fe3174f64b0936c512ee08f48; no voters:
I20260925 03:02:51.857303 23632 leader_election.cc:288] T ef74cafe53e34e1dace8b02d73693521 P fa551b23283b4034b256ba25acec541b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b66c1d3fe3174f64b0936c512ee08f48 (127.22.193.3:42621), 2edab657a33a4a0abfbd3262b1668483 (127.22.193.2:35431)
I20260925 03:02:51.856747 23637 raft_consensus.cc:2781] T ef74cafe53e34e1dace8b02d73693521 P 2edab657a33a4a0abfbd3262b1668483 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 03:02:51.858338 23637 raft_consensus.cc:482] T ef74cafe53e34e1dace8b02d73693521 P 2edab657a33a4a0abfbd3262b1668483 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:02:51.858968 23637 raft_consensus.cc:3037] T ef74cafe53e34e1dace8b02d73693521 P 2edab657a33a4a0abfbd3262b1668483 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:02:51.868834 23628 tablet_bootstrap.cc:654] T 9de619025d19432c8df31a19bb6fc446 P 2edab657a33a4a0abfbd3262b1668483: Neither blocks nor log segments found. Creating new log.
I20260925 03:02:51.869189 23635 raft_consensus.cc:482] T 2baf62cd63564f4ba08b201e40e19d55 P 2edab657a33a4a0abfbd3262b1668483 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:02:51.869705 23635 raft_consensus.cc:504] T 2baf62cd63564f4ba08b201e40e19d55 P 2edab657a33a4a0abfbd3262b1668483 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:02:51.870774 23637 raft_consensus.cc:504] T ef74cafe53e34e1dace8b02d73693521 P 2edab657a33a4a0abfbd3262b1668483 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } } peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } }
I20260925 03:02:51.872210 23635 leader_election.cc:288] T 2baf62cd63564f4ba08b201e40e19d55 P 2edab657a33a4a0abfbd3262b1668483 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa551b23283b4034b256ba25acec541b (127.22.193.1:42181), b66c1d3fe3174f64b0936c512ee08f48 (127.22.193.3:42621)
I20260925 03:02:51.874320 23435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2baf62cd63564f4ba08b201e40e19d55" candidate_uuid: "2edab657a33a4a0abfbd3262b1668483" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa551b23283b4034b256ba25acec541b" is_pre_election: true
I20260925 03:02:51.876260 23435 raft_consensus.cc:2454] T 2baf62cd63564f4ba08b201e40e19d55 P fa551b23283b4034b256ba25acec541b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2edab657a33a4a0abfbd3262b1668483 in term 0.
I20260925 03:02:51.877220 23637 leader_election.cc:288] T ef74cafe53e34e1dace8b02d73693521 P 2edab657a33a4a0abfbd3262b1668483 [CANDIDATE]: Term 1 election: Requested vote from peers b66c1d3fe3174f64b0936c512ee08f48 (127.22.193.3:42621), fa551b23283b4034b256ba25acec541b (127.22.193.1:42181)
I20260925 03:02:51.876992 23584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2baf62cd63564f4ba08b201e40e19d55" candidate_uuid: "2edab657a33a4a0abfbd3262b1668483" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b66c1d3fe3174f64b0936c512ee08f48" is_pre_election: true
I20260925 03:02:51.877947 23473 leader_election.cc:302] T 2baf62cd63564f4ba08b201e40e19d55 P 2edab657a33a4a0abfbd3262b1668483 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2edab657a33a4a0abfbd3262b1668483, fa551b23283b4034b256ba25acec541b; no voters:
I20260925 03:02:51.880151 23583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef74cafe53e34e1dace8b02d73693521" candidate_uuid: "2edab657a33a4a0abfbd3262b1668483" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b66c1d3fe3174f64b0936c512ee08f48"
I20260925 03:02:51.883379 23583 raft_consensus.cc:3037] T ef74cafe53e34e1dace8b02d73693521 P b66c1d3fe3174f64b0936c512ee08f48 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:02:51.884292 23637 raft_consensus.cc:2781] T 2baf62cd63564f4ba08b201e40e19d55 P 2edab657a33a4a0abfbd3262b1668483 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 03:02:51.884644 23637 raft_consensus.cc:482] T 2baf62cd63564f4ba08b201e40e19d55 P 2edab657a33a4a0abfbd3262b1668483 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:02:51.884078 23584 raft_consensus.cc:2454] T 2baf62cd63564f4ba08b201e40e19d55 P b66c1d3fe3174f64b0936c512ee08f48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2edab657a33a4a0abfbd3262b1668483 in term 0.
I20260925 03:02:51.886996 23435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef74cafe53e34e1dace8b02d73693521" candidate_uuid: "2edab657a33a4a0abfbd3262b1668483" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa551b23283b4034b256ba25acec541b"
I20260925 03:02:51.887912 23637 raft_consensus.cc:3037] T 2baf62cd63564f4ba08b201e40e19d55 P 2edab657a33a4a0abfbd3262b1668483 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:02:51.888900 23435 raft_consensus.cc:3037] T ef74cafe53e34e1dace8b02d73693521 P fa551b23283b4034b256ba25acec541b [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:02:51.896466 23435 raft_consensus.cc:2454] T ef74cafe53e34e1dace8b02d73693521 P fa551b23283b4034b256ba25acec541b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2edab657a33a4a0abfbd3262b1668483 in term 1.
I20260925 03:02:51.897109 23629 tablet_bootstrap.cc:654] T d01220d66a8c4d34ac86fc3559fa5165 P fa551b23283b4034b256ba25acec541b: Neither blocks nor log segments found. Creating new log.
I20260925 03:02:51.897681 23473 leader_election.cc:302] T ef74cafe53e34e1dace8b02d73693521 P 2edab657a33a4a0abfbd3262b1668483 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2edab657a33a4a0abfbd3262b1668483, fa551b23283b4034b256ba25acec541b; no voters:
I20260925 03:02:51.898725 23635 raft_consensus.cc:2781] T ef74cafe53e34e1dace8b02d73693521 P 2edab657a33a4a0abfbd3262b1668483 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:02:51.898386 23627 tablet_bootstrap.cc:654] T fb663a8b564c494f98f56845f8a05d6c P b66c1d3fe3174f64b0936c512ee08f48: Neither blocks nor log segments found. Creating new log.
I20260925 03:02:51.904505 23583 raft_consensus.cc:2454] T ef74cafe53e34e1dace8b02d73693521 P b66c1d3fe3174f64b0936c512ee08f48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2edab657a33a4a0abfbd3262b1668483 in term 1.
I20260925 03:02:51.914249 23635 raft_consensus.cc:686] T ef74cafe53e34e1dace8b02d73693521 P 2edab657a33a4a0abfbd3262b1668483 [term 1 LEADER]: Becoming Leader. State: Replica: 2edab657a33a4a0abfbd3262b1668483, State: Running, Role: LEADER
I20260925 03:02:51.920959 23635 consensus_queue.cc:237] T ef74cafe53e34e1dace8b02d73693521 P 2edab657a33a4a0abfbd3262b1668483 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } } peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } }
I20260925 03:02:51.925819 23637 raft_consensus.cc:504] T 2baf62cd63564f4ba08b201e40e19d55 P 2edab657a33a4a0abfbd3262b1668483 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:02:51.933470 23660 raft_consensus.cc:482] T d01220d66a8c4d34ac86fc3559fa5165 P 2edab657a33a4a0abfbd3262b1668483 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:02:51.940621 23435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2baf62cd63564f4ba08b201e40e19d55" candidate_uuid: "2edab657a33a4a0abfbd3262b1668483" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa551b23283b4034b256ba25acec541b"
I20260925 03:02:51.941416 23435 raft_consensus.cc:3037] T 2baf62cd63564f4ba08b201e40e19d55 P fa551b23283b4034b256ba25acec541b [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:02:51.941628 23637 leader_election.cc:288] T 2baf62cd63564f4ba08b201e40e19d55 P 2edab657a33a4a0abfbd3262b1668483 [CANDIDATE]: Term 1 election: Requested vote from peers fa551b23283b4034b256ba25acec541b (127.22.193.1:42181), b66c1d3fe3174f64b0936c512ee08f48 (127.22.193.3:42621)
I20260925 03:02:51.942018 23628 tablet_bootstrap.cc:492] T 9de619025d19432c8df31a19bb6fc446 P 2edab657a33a4a0abfbd3262b1668483: No bootstrap required, opened a new log
I20260925 03:02:51.942528 23628 ts_tablet_manager.cc:1397] T 9de619025d19432c8df31a19bb6fc446 P 2edab657a33a4a0abfbd3262b1668483: Time spent bootstrapping tablet: real 0.110s user 0.012s sys 0.006s
I20260925 03:02:51.944589 23583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2baf62cd63564f4ba08b201e40e19d55" candidate_uuid: "2edab657a33a4a0abfbd3262b1668483" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b66c1d3fe3174f64b0936c512ee08f48"
I20260925 03:02:51.945242 23583 raft_consensus.cc:3037] T 2baf62cd63564f4ba08b201e40e19d55 P b66c1d3fe3174f64b0936c512ee08f48 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:02:51.943663 23660 raft_consensus.cc:504] T d01220d66a8c4d34ac86fc3559fa5165 P 2edab657a33a4a0abfbd3262b1668483 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:02:51.951632 23584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef74cafe53e34e1dace8b02d73693521" candidate_uuid: "fa551b23283b4034b256ba25acec541b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b66c1d3fe3174f64b0936c512ee08f48" is_pre_election: true
I20260925 03:02:51.954393 23584 raft_consensus.cc:2379] T ef74cafe53e34e1dace8b02d73693521 P b66c1d3fe3174f64b0936c512ee08f48 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fa551b23283b4034b256ba25acec541b in current term 1: Already voted for candidate 2edab657a33a4a0abfbd3262b1668483 in this term.
I20260925 03:02:51.954617 23629 tablet_bootstrap.cc:492] T d01220d66a8c4d34ac86fc3559fa5165 P fa551b23283b4034b256ba25acec541b: No bootstrap required, opened a new log
I20260925 03:02:51.955121 23629 ts_tablet_manager.cc:1397] T d01220d66a8c4d34ac86fc3559fa5165 P fa551b23283b4034b256ba25acec541b: Time spent bootstrapping tablet: real 0.115s user 0.007s sys 0.016s
I20260925 03:02:51.955489 23435 raft_consensus.cc:2454] T 2baf62cd63564f4ba08b201e40e19d55 P fa551b23283b4034b256ba25acec541b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2edab657a33a4a0abfbd3262b1668483 in term 1.
I20260925 03:02:51.959453 23473 leader_election.cc:302] T 2baf62cd63564f4ba08b201e40e19d55 P 2edab657a33a4a0abfbd3262b1668483 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2edab657a33a4a0abfbd3262b1668483, fa551b23283b4034b256ba25acec541b; no voters:
I20260925 03:02:51.958981 23332 catalog_manager.cc:5495] T ef74cafe53e34e1dace8b02d73693521 P 2edab657a33a4a0abfbd3262b1668483 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2edab657a33a4a0abfbd3262b1668483 (127.22.193.2). New cstate: current_term: 1 leader_uuid: "2edab657a33a4a0abfbd3262b1668483" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } health_report { overall_health: HEALTHY } } }
I20260925 03:02:51.959942 23583 raft_consensus.cc:2454] T 2baf62cd63564f4ba08b201e40e19d55 P b66c1d3fe3174f64b0936c512ee08f48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2edab657a33a4a0abfbd3262b1668483 in term 1.
I20260925 03:02:51.960410 23635 raft_consensus.cc:2781] T 2baf62cd63564f4ba08b201e40e19d55 P 2edab657a33a4a0abfbd3262b1668483 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:02:51.961187 23635 raft_consensus.cc:686] T 2baf62cd63564f4ba08b201e40e19d55 P 2edab657a33a4a0abfbd3262b1668483 [term 1 LEADER]: Becoming Leader. State: Replica: 2edab657a33a4a0abfbd3262b1668483, State: Running, Role: LEADER
I20260925 03:02:51.962229 23635 consensus_queue.cc:237] T 2baf62cd63564f4ba08b201e40e19d55 P 2edab657a33a4a0abfbd3262b1668483 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:02:51.953402 23628 raft_consensus.cc:348] T 9de619025d19432c8df31a19bb6fc446 P 2edab657a33a4a0abfbd3262b1668483 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:02:51.969594 23628 raft_consensus.cc:374] T 9de619025d19432c8df31a19bb6fc446 P 2edab657a33a4a0abfbd3262b1668483 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:02:51.974206 23628 raft_consensus.cc:729] T 9de619025d19432c8df31a19bb6fc446 P 2edab657a33a4a0abfbd3262b1668483 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2edab657a33a4a0abfbd3262b1668483, State: Initialized, Role: FOLLOWER
I20260925 03:02:51.975222 23510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef74cafe53e34e1dace8b02d73693521" candidate_uuid: "fa551b23283b4034b256ba25acec541b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2edab657a33a4a0abfbd3262b1668483" is_pre_election: true
I20260925 03:02:51.975553 23629 raft_consensus.cc:348] T d01220d66a8c4d34ac86fc3559fa5165 P fa551b23283b4034b256ba25acec541b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:02:51.982353 23399 leader_election.cc:302] T ef74cafe53e34e1dace8b02d73693521 P fa551b23283b4034b256ba25acec541b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fa551b23283b4034b256ba25acec541b; no voters: 2edab657a33a4a0abfbd3262b1668483, b66c1d3fe3174f64b0936c512ee08f48
I20260925 03:02:51.984241 23632 raft_consensus.cc:2726] T ef74cafe53e34e1dace8b02d73693521 P fa551b23283b4034b256ba25acec541b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 03:02:51.983031 23628 consensus_queue.cc:260] T 9de619025d19432c8df31a19bb6fc446 P 2edab657a33a4a0abfbd3262b1668483 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
W20260925 03:02:51.984633 23380 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260925 03:02:51.985450 23660 leader_election.cc:288] T d01220d66a8c4d34ac86fc3559fa5165 P 2edab657a33a4a0abfbd3262b1668483 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa551b23283b4034b256ba25acec541b (127.22.193.1:42181), b66c1d3fe3174f64b0936c512ee08f48 (127.22.193.3:42621)
I20260925 03:02:51.986562 23583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d01220d66a8c4d34ac86fc3559fa5165" candidate_uuid: "2edab657a33a4a0abfbd3262b1668483" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b66c1d3fe3174f64b0936c512ee08f48" is_pre_election: true
I20260925 03:02:51.987233 23435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d01220d66a8c4d34ac86fc3559fa5165" candidate_uuid: "2edab657a33a4a0abfbd3262b1668483" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa551b23283b4034b256ba25acec541b" is_pre_election: true
I20260925 03:02:51.984001 23629 raft_consensus.cc:374] T d01220d66a8c4d34ac86fc3559fa5165 P fa551b23283b4034b256ba25acec541b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:02:51.988953 23629 raft_consensus.cc:729] T d01220d66a8c4d34ac86fc3559fa5165 P fa551b23283b4034b256ba25acec541b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa551b23283b4034b256ba25acec541b, State: Initialized, Role: FOLLOWER
I20260925 03:02:51.988503 23583 raft_consensus.cc:2454] T d01220d66a8c4d34ac86fc3559fa5165 P b66c1d3fe3174f64b0936c512ee08f48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2edab657a33a4a0abfbd3262b1668483 in term 0.
I20260925 03:02:51.990470 23627 tablet_bootstrap.cc:492] T fb663a8b564c494f98f56845f8a05d6c P b66c1d3fe3174f64b0936c512ee08f48: No bootstrap required, opened a new log
I20260925 03:02:51.991950 23629 consensus_queue.cc:260] T d01220d66a8c4d34ac86fc3559fa5165 P fa551b23283b4034b256ba25acec541b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:02:51.992345 23473 leader_election.cc:302] T d01220d66a8c4d34ac86fc3559fa5165 P 2edab657a33a4a0abfbd3262b1668483 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2edab657a33a4a0abfbd3262b1668483, b66c1d3fe3174f64b0936c512ee08f48; no voters:
I20260925 03:02:51.990445 23332 catalog_manager.cc:5495] T 2baf62cd63564f4ba08b201e40e19d55 P 2edab657a33a4a0abfbd3262b1668483 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2edab657a33a4a0abfbd3262b1668483 (127.22.193.2). New cstate: current_term: 1 leader_uuid: "2edab657a33a4a0abfbd3262b1668483" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } health_report { overall_health: UNKNOWN } } }
I20260925 03:02:51.993088 23435 raft_consensus.cc:2454] T d01220d66a8c4d34ac86fc3559fa5165 P fa551b23283b4034b256ba25acec541b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2edab657a33a4a0abfbd3262b1668483 in term 0.
I20260925 03:02:51.992365 23627 ts_tablet_manager.cc:1397] T fb663a8b564c494f98f56845f8a05d6c P b66c1d3fe3174f64b0936c512ee08f48: Time spent bootstrapping tablet: real 0.138s user 0.009s sys 0.018s
I20260925 03:02:51.993577 23628 ts_tablet_manager.cc:1428] T 9de619025d19432c8df31a19bb6fc446 P 2edab657a33a4a0abfbd3262b1668483: Time spent starting tablet: real 0.051s user 0.000s sys 0.011s
I20260925 03:02:51.994594 23628 tablet_bootstrap.cc:492] T fb663a8b564c494f98f56845f8a05d6c P 2edab657a33a4a0abfbd3262b1668483: Bootstrap starting.
I20260925 03:02:51.995362 23632 raft_consensus.cc:482] T 832e06098a3e499f82d53853e26235cf P fa551b23283b4034b256ba25acec541b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:02:51.996435 23627 raft_consensus.cc:348] T fb663a8b564c494f98f56845f8a05d6c P b66c1d3fe3174f64b0936c512ee08f48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:02:51.996091 23632 raft_consensus.cc:504] T 832e06098a3e499f82d53853e26235cf P fa551b23283b4034b256ba25acec541b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:02:51.998510 23632 leader_election.cc:288] T 832e06098a3e499f82d53853e26235cf P fa551b23283b4034b256ba25acec541b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2edab657a33a4a0abfbd3262b1668483 (127.22.193.2:35431), b66c1d3fe3174f64b0936c512ee08f48 (127.22.193.3:42621)
I20260925 03:02:51.998874 23629 ts_tablet_manager.cc:1428] T d01220d66a8c4d34ac86fc3559fa5165 P fa551b23283b4034b256ba25acec541b: Time spent starting tablet: real 0.043s user 0.008s sys 0.002s
I20260925 03:02:51.999032 23627 raft_consensus.cc:374] T fb663a8b564c494f98f56845f8a05d6c P b66c1d3fe3174f64b0936c512ee08f48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:02:51.999490 23627 raft_consensus.cc:729] T fb663a8b564c494f98f56845f8a05d6c P b66c1d3fe3174f64b0936c512ee08f48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b66c1d3fe3174f64b0936c512ee08f48, State: Initialized, Role: FOLLOWER
I20260925 03:02:51.999541 23660 raft_consensus.cc:2781] T d01220d66a8c4d34ac86fc3559fa5165 P 2edab657a33a4a0abfbd3262b1668483 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 03:02:51.999619 23583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "832e06098a3e499f82d53853e26235cf" candidate_uuid: "fa551b23283b4034b256ba25acec541b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b66c1d3fe3174f64b0936c512ee08f48" is_pre_election: true
I20260925 03:02:52.000058 23660 raft_consensus.cc:482] T d01220d66a8c4d34ac86fc3559fa5165 P 2edab657a33a4a0abfbd3262b1668483 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:02:52.000432 23660 raft_consensus.cc:3037] T d01220d66a8c4d34ac86fc3559fa5165 P 2edab657a33a4a0abfbd3262b1668483 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:02:52.000583 23627 consensus_queue.cc:260] T fb663a8b564c494f98f56845f8a05d6c P b66c1d3fe3174f64b0936c512ee08f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
W20260925 03:02:52.001299 23398 leader_election.cc:341] T 832e06098a3e499f82d53853e26235cf P fa551b23283b4034b256ba25acec541b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b66c1d3fe3174f64b0936c512ee08f48 (127.22.193.3:42621): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:02:52.003921 23627 ts_tablet_manager.cc:1428] T fb663a8b564c494f98f56845f8a05d6c P b66c1d3fe3174f64b0936c512ee08f48: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20260925 03:02:52.009370 23629 tablet_bootstrap.cc:492] T 138dfa413a144a0e8911a2b402857cb4 P fa551b23283b4034b256ba25acec541b: Bootstrap starting.
I20260925 03:02:52.009804 23627 tablet_bootstrap.cc:492] T c30e9a1092fe4d99994480aab8863cf6 P b66c1d3fe3174f64b0936c512ee08f48: Bootstrap starting.
I20260925 03:02:52.011518 23628 tablet_bootstrap.cc:654] T fb663a8b564c494f98f56845f8a05d6c P 2edab657a33a4a0abfbd3262b1668483: Neither blocks nor log segments found. Creating new log.
I20260925 03:02:52.011703 23634 raft_consensus.cc:482] T fb663a8b564c494f98f56845f8a05d6c P b66c1d3fe3174f64b0936c512ee08f48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:02:52.012378 23634 raft_consensus.cc:504] T fb663a8b564c494f98f56845f8a05d6c P b66c1d3fe3174f64b0936c512ee08f48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:02:52.014801 23634 leader_election.cc:288] T fb663a8b564c494f98f56845f8a05d6c P b66c1d3fe3174f64b0936c512ee08f48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa551b23283b4034b256ba25acec541b (127.22.193.1:42181), 2edab657a33a4a0abfbd3262b1668483 (127.22.193.2:35431)
I20260925 03:02:52.017427 23510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb663a8b564c494f98f56845f8a05d6c" candidate_uuid: "b66c1d3fe3174f64b0936c512ee08f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2edab657a33a4a0abfbd3262b1668483" is_pre_election: true
W20260925 03:02:52.020083 23548 leader_election.cc:341] T fb663a8b564c494f98f56845f8a05d6c P b66c1d3fe3174f64b0936c512ee08f48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2edab657a33a4a0abfbd3262b1668483 (127.22.193.2:35431): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:02:52.018294 23435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb663a8b564c494f98f56845f8a05d6c" candidate_uuid: "b66c1d3fe3174f64b0936c512ee08f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa551b23283b4034b256ba25acec541b" is_pre_election: true
W20260925 03:02:52.022647 23547 leader_election.cc:341] T fb663a8b564c494f98f56845f8a05d6c P b66c1d3fe3174f64b0936c512ee08f48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fa551b23283b4034b256ba25acec541b (127.22.193.1:42181): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:02:52.023104 23547 leader_election.cc:302] T fb663a8b564c494f98f56845f8a05d6c P b66c1d3fe3174f64b0936c512ee08f48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b66c1d3fe3174f64b0936c512ee08f48; no voters: 2edab657a33a4a0abfbd3262b1668483, fa551b23283b4034b256ba25acec541b
I20260925 03:02:52.023509 23660 raft_consensus.cc:504] T d01220d66a8c4d34ac86fc3559fa5165 P 2edab657a33a4a0abfbd3262b1668483 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:02:52.024314 23634 raft_consensus.cc:2726] T fb663a8b564c494f98f56845f8a05d6c P b66c1d3fe3174f64b0936c512ee08f48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 03:02:52.028877 23510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "832e06098a3e499f82d53853e26235cf" candidate_uuid: "fa551b23283b4034b256ba25acec541b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2edab657a33a4a0abfbd3262b1668483" is_pre_election: true
I20260925 03:02:52.028178 23628 tablet_bootstrap.cc:492] T fb663a8b564c494f98f56845f8a05d6c P 2edab657a33a4a0abfbd3262b1668483: No bootstrap required, opened a new log
I20260925 03:02:52.032819 23435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d01220d66a8c4d34ac86fc3559fa5165" candidate_uuid: "2edab657a33a4a0abfbd3262b1668483" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa551b23283b4034b256ba25acec541b"
I20260925 03:02:52.033138 23583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d01220d66a8c4d34ac86fc3559fa5165" candidate_uuid: "2edab657a33a4a0abfbd3262b1668483" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b66c1d3fe3174f64b0936c512ee08f48"
I20260925 03:02:52.033438 23435 raft_consensus.cc:3037] T d01220d66a8c4d34ac86fc3559fa5165 P fa551b23283b4034b256ba25acec541b [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:02:52.033502 23628 ts_tablet_manager.cc:1397] T fb663a8b564c494f98f56845f8a05d6c P 2edab657a33a4a0abfbd3262b1668483: Time spent bootstrapping tablet: real 0.039s user 0.014s sys 0.006s
I20260925 03:02:52.033775 23583 raft_consensus.cc:3037] T d01220d66a8c4d34ac86fc3559fa5165 P b66c1d3fe3174f64b0936c512ee08f48 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:02:52.036818 23628 raft_consensus.cc:348] T fb663a8b564c494f98f56845f8a05d6c P 2edab657a33a4a0abfbd3262b1668483 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:02:52.031872 23660 leader_election.cc:288] T d01220d66a8c4d34ac86fc3559fa5165 P 2edab657a33a4a0abfbd3262b1668483 [CANDIDATE]: Term 1 election: Requested vote from peers fa551b23283b4034b256ba25acec541b (127.22.193.1:42181), b66c1d3fe3174f64b0936c512ee08f48 (127.22.193.3:42621)
I20260925 03:02:52.037744 23629 tablet_bootstrap.cc:654] T 138dfa413a144a0e8911a2b402857cb4 P fa551b23283b4034b256ba25acec541b: Neither blocks nor log segments found. Creating new log.
I20260925 03:02:52.041456 23583 raft_consensus.cc:2454] T d01220d66a8c4d34ac86fc3559fa5165 P b66c1d3fe3174f64b0936c512ee08f48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2edab657a33a4a0abfbd3262b1668483 in term 1.
W20260925 03:02:52.030428 23399 leader_election.cc:341] T 832e06098a3e499f82d53853e26235cf P fa551b23283b4034b256ba25acec541b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2edab657a33a4a0abfbd3262b1668483 (127.22.193.2:35431): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:02:52.040331 23435 raft_consensus.cc:2454] T d01220d66a8c4d34ac86fc3559fa5165 P fa551b23283b4034b256ba25acec541b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2edab657a33a4a0abfbd3262b1668483 in term 1.
I20260925 03:02:52.040477 23627 tablet_bootstrap.cc:654] T c30e9a1092fe4d99994480aab8863cf6 P b66c1d3fe3174f64b0936c512ee08f48: Neither blocks nor log segments found. Creating new log.
I20260925 03:02:52.043270 23473 leader_election.cc:302] T d01220d66a8c4d34ac86fc3559fa5165 P 2edab657a33a4a0abfbd3262b1668483 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2edab657a33a4a0abfbd3262b1668483, b66c1d3fe3174f64b0936c512ee08f48; no voters:
I20260925 03:02:52.042867 23628 raft_consensus.cc:374] T fb663a8b564c494f98f56845f8a05d6c P 2edab657a33a4a0abfbd3262b1668483 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:02:52.044968 23628 raft_consensus.cc:729] T fb663a8b564c494f98f56845f8a05d6c P 2edab657a33a4a0abfbd3262b1668483 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2edab657a33a4a0abfbd3262b1668483, State: Initialized, Role: FOLLOWER
I20260925 03:02:52.045042 23660 raft_consensus.cc:2781] T d01220d66a8c4d34ac86fc3559fa5165 P 2edab657a33a4a0abfbd3262b1668483 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:02:52.042672 23399 leader_election.cc:302] T 832e06098a3e499f82d53853e26235cf P fa551b23283b4034b256ba25acec541b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fa551b23283b4034b256ba25acec541b; no voters: 2edab657a33a4a0abfbd3262b1668483, b66c1d3fe3174f64b0936c512ee08f48
I20260925 03:02:52.045552 23660 raft_consensus.cc:686] T d01220d66a8c4d34ac86fc3559fa5165 P 2edab657a33a4a0abfbd3262b1668483 [term 1 LEADER]: Becoming Leader. State: Replica: 2edab657a33a4a0abfbd3262b1668483, State: Running, Role: LEADER
I20260925 03:02:52.046857 23665 raft_consensus.cc:2726] T 832e06098a3e499f82d53853e26235cf P fa551b23283b4034b256ba25acec541b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 03:02:52.046936 23660 consensus_queue.cc:237] T d01220d66a8c4d34ac86fc3559fa5165 P 2edab657a33a4a0abfbd3262b1668483 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:02:52.046083 23628 consensus_queue.cc:260] T fb663a8b564c494f98f56845f8a05d6c P 2edab657a33a4a0abfbd3262b1668483 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:02:52.052575 23628 ts_tablet_manager.cc:1428] T fb663a8b564c494f98f56845f8a05d6c P 2edab657a33a4a0abfbd3262b1668483: Time spent starting tablet: real 0.019s user 0.006s sys 0.000s
I20260925 03:02:52.054497 23628 tablet_bootstrap.cc:492] T c30e9a1092fe4d99994480aab8863cf6 P 2edab657a33a4a0abfbd3262b1668483: Bootstrap starting.
I20260925 03:02:52.064851 23627 tablet_bootstrap.cc:492] T c30e9a1092fe4d99994480aab8863cf6 P b66c1d3fe3174f64b0936c512ee08f48: No bootstrap required, opened a new log
I20260925 03:02:52.063242 23628 tablet_bootstrap.cc:654] T c30e9a1092fe4d99994480aab8863cf6 P 2edab657a33a4a0abfbd3262b1668483: Neither blocks nor log segments found. Creating new log.
I20260925 03:02:52.065290 23629 tablet_bootstrap.cc:492] T 138dfa413a144a0e8911a2b402857cb4 P fa551b23283b4034b256ba25acec541b: No bootstrap required, opened a new log
I20260925 03:02:52.065415 23627 ts_tablet_manager.cc:1397] T c30e9a1092fe4d99994480aab8863cf6 P b66c1d3fe3174f64b0936c512ee08f48: Time spent bootstrapping tablet: real 0.056s user 0.010s sys 0.004s
I20260925 03:02:52.065749 23629 ts_tablet_manager.cc:1397] T 138dfa413a144a0e8911a2b402857cb4 P fa551b23283b4034b256ba25acec541b: Time spent bootstrapping tablet: real 0.057s user 0.008s sys 0.010s
I20260925 03:02:52.071563 23627 raft_consensus.cc:348] T c30e9a1092fe4d99994480aab8863cf6 P b66c1d3fe3174f64b0936c512ee08f48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:02:52.073792 23627 raft_consensus.cc:374] T c30e9a1092fe4d99994480aab8863cf6 P b66c1d3fe3174f64b0936c512ee08f48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:02:52.074316 23627 raft_consensus.cc:729] T c30e9a1092fe4d99994480aab8863cf6 P b66c1d3fe3174f64b0936c512ee08f48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b66c1d3fe3174f64b0936c512ee08f48, State: Initialized, Role: FOLLOWER
I20260925 03:02:52.074692 23629 raft_consensus.cc:348] T 138dfa413a144a0e8911a2b402857cb4 P fa551b23283b4034b256ba25acec541b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:02:52.074772 23332 catalog_manager.cc:5495] T d01220d66a8c4d34ac86fc3559fa5165 P 2edab657a33a4a0abfbd3262b1668483 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2edab657a33a4a0abfbd3262b1668483 (127.22.193.2). New cstate: current_term: 1 leader_uuid: "2edab657a33a4a0abfbd3262b1668483" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } health_report { overall_health: UNKNOWN } } }
I20260925 03:02:52.076406 23629 raft_consensus.cc:374] T 138dfa413a144a0e8911a2b402857cb4 P fa551b23283b4034b256ba25acec541b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:02:52.077003 23629 raft_consensus.cc:729] T 138dfa413a144a0e8911a2b402857cb4 P fa551b23283b4034b256ba25acec541b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa551b23283b4034b256ba25acec541b, State: Initialized, Role: FOLLOWER
I20260925 03:02:52.075453 23627 consensus_queue.cc:260] T c30e9a1092fe4d99994480aab8863cf6 P b66c1d3fe3174f64b0936c512ee08f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:02:52.077836 23629 consensus_queue.cc:260] T 138dfa413a144a0e8911a2b402857cb4 P fa551b23283b4034b256ba25acec541b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:02:52.080202 23627 ts_tablet_manager.cc:1428] T c30e9a1092fe4d99994480aab8863cf6 P b66c1d3fe3174f64b0936c512ee08f48: Time spent starting tablet: real 0.014s user 0.009s sys 0.000s
I20260925 03:02:52.081687 23627 tablet_bootstrap.cc:492] T 138dfa413a144a0e8911a2b402857cb4 P b66c1d3fe3174f64b0936c512ee08f48: Bootstrap starting.
I20260925 03:02:52.085364 23628 tablet_bootstrap.cc:492] T c30e9a1092fe4d99994480aab8863cf6 P 2edab657a33a4a0abfbd3262b1668483: No bootstrap required, opened a new log
I20260925 03:02:52.085913 23628 ts_tablet_manager.cc:1397] T c30e9a1092fe4d99994480aab8863cf6 P 2edab657a33a4a0abfbd3262b1668483: Time spent bootstrapping tablet: real 0.032s user 0.010s sys 0.004s
I20260925 03:02:52.087589 23629 ts_tablet_manager.cc:1428] T 138dfa413a144a0e8911a2b402857cb4 P fa551b23283b4034b256ba25acec541b: Time spent starting tablet: real 0.021s user 0.000s sys 0.010s
I20260925 03:02:52.089247 23629 tablet_bootstrap.cc:492] T fb663a8b564c494f98f56845f8a05d6c P fa551b23283b4034b256ba25acec541b: Bootstrap starting.
I20260925 03:02:52.093426 23627 tablet_bootstrap.cc:654] T 138dfa413a144a0e8911a2b402857cb4 P b66c1d3fe3174f64b0936c512ee08f48: Neither blocks nor log segments found. Creating new log.
I20260925 03:02:52.098690 23629 tablet_bootstrap.cc:654] T fb663a8b564c494f98f56845f8a05d6c P fa551b23283b4034b256ba25acec541b: Neither blocks nor log segments found. Creating new log.
I20260925 03:02:52.103410 23628 raft_consensus.cc:348] T c30e9a1092fe4d99994480aab8863cf6 P 2edab657a33a4a0abfbd3262b1668483 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:02:52.104270 23628 raft_consensus.cc:374] T c30e9a1092fe4d99994480aab8863cf6 P 2edab657a33a4a0abfbd3262b1668483 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:02:52.104579 23628 raft_consensus.cc:729] T c30e9a1092fe4d99994480aab8863cf6 P 2edab657a33a4a0abfbd3262b1668483 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2edab657a33a4a0abfbd3262b1668483, State: Initialized, Role: FOLLOWER
I20260925 03:02:52.105377 23628 consensus_queue.cc:260] T c30e9a1092fe4d99994480aab8863cf6 P 2edab657a33a4a0abfbd3262b1668483 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:02:52.107668 23628 ts_tablet_manager.cc:1428] T c30e9a1092fe4d99994480aab8863cf6 P 2edab657a33a4a0abfbd3262b1668483: Time spent starting tablet: real 0.021s user 0.005s sys 0.001s
I20260925 03:02:52.108631 23628 tablet_bootstrap.cc:492] T 832e06098a3e499f82d53853e26235cf P 2edab657a33a4a0abfbd3262b1668483: Bootstrap starting.
I20260925 03:02:52.110867 23627 tablet_bootstrap.cc:492] T 138dfa413a144a0e8911a2b402857cb4 P b66c1d3fe3174f64b0936c512ee08f48: No bootstrap required, opened a new log
I20260925 03:02:52.111259 23627 ts_tablet_manager.cc:1397] T 138dfa413a144a0e8911a2b402857cb4 P b66c1d3fe3174f64b0936c512ee08f48: Time spent bootstrapping tablet: real 0.030s user 0.010s sys 0.009s
I20260925 03:02:52.114316 23627 raft_consensus.cc:348] T 138dfa413a144a0e8911a2b402857cb4 P b66c1d3fe3174f64b0936c512ee08f48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:02:52.115089 23627 raft_consensus.cc:374] T 138dfa413a144a0e8911a2b402857cb4 P b66c1d3fe3174f64b0936c512ee08f48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:02:52.115152 23629 tablet_bootstrap.cc:492] T fb663a8b564c494f98f56845f8a05d6c P fa551b23283b4034b256ba25acec541b: No bootstrap required, opened a new log
I20260925 03:02:52.115432 23627 raft_consensus.cc:729] T 138dfa413a144a0e8911a2b402857cb4 P b66c1d3fe3174f64b0936c512ee08f48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b66c1d3fe3174f64b0936c512ee08f48, State: Initialized, Role: FOLLOWER
I20260925 03:02:52.115707 23629 ts_tablet_manager.cc:1397] T fb663a8b564c494f98f56845f8a05d6c P fa551b23283b4034b256ba25acec541b: Time spent bootstrapping tablet: real 0.027s user 0.014s sys 0.002s
I20260925 03:02:52.116392 23627 consensus_queue.cc:260] T 138dfa413a144a0e8911a2b402857cb4 P b66c1d3fe3174f64b0936c512ee08f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:02:52.118790 23627 ts_tablet_manager.cc:1428] T 138dfa413a144a0e8911a2b402857cb4 P b66c1d3fe3174f64b0936c512ee08f48: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260925 03:02:52.118659 23629 raft_consensus.cc:348] T fb663a8b564c494f98f56845f8a05d6c P fa551b23283b4034b256ba25acec541b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:02:52.119553 23629 raft_consensus.cc:374] T fb663a8b564c494f98f56845f8a05d6c P fa551b23283b4034b256ba25acec541b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:02:52.119724 23627 tablet_bootstrap.cc:492] T 43de311d912b475eb01e1f7c9a65412a P b66c1d3fe3174f64b0936c512ee08f48: Bootstrap starting.
I20260925 03:02:52.120028 23629 raft_consensus.cc:729] T fb663a8b564c494f98f56845f8a05d6c P fa551b23283b4034b256ba25acec541b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa551b23283b4034b256ba25acec541b, State: Initialized, Role: FOLLOWER
I20260925 03:02:52.121044 23629 consensus_queue.cc:260] T fb663a8b564c494f98f56845f8a05d6c P fa551b23283b4034b256ba25acec541b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:02:52.120235 23628 tablet_bootstrap.cc:654] T 832e06098a3e499f82d53853e26235cf P 2edab657a33a4a0abfbd3262b1668483: Neither blocks nor log segments found. Creating new log.
I20260925 03:02:52.125276 23629 ts_tablet_manager.cc:1428] T fb663a8b564c494f98f56845f8a05d6c P fa551b23283b4034b256ba25acec541b: Time spent starting tablet: real 0.009s user 0.003s sys 0.003s
I20260925 03:02:52.126276 23629 tablet_bootstrap.cc:492] T c30e9a1092fe4d99994480aab8863cf6 P fa551b23283b4034b256ba25acec541b: Bootstrap starting.
I20260925 03:02:52.130977 23627 tablet_bootstrap.cc:654] T 43de311d912b475eb01e1f7c9a65412a P b66c1d3fe3174f64b0936c512ee08f48: Neither blocks nor log segments found. Creating new log.
I20260925 03:02:52.136557 23628 tablet_bootstrap.cc:492] T 832e06098a3e499f82d53853e26235cf P 2edab657a33a4a0abfbd3262b1668483: No bootstrap required, opened a new log
I20260925 03:02:52.137089 23628 ts_tablet_manager.cc:1397] T 832e06098a3e499f82d53853e26235cf P 2edab657a33a4a0abfbd3262b1668483: Time spent bootstrapping tablet: real 0.029s user 0.007s sys 0.008s
I20260925 03:02:52.141569 23629 tablet_bootstrap.cc:654] T c30e9a1092fe4d99994480aab8863cf6 P fa551b23283b4034b256ba25acec541b: Neither blocks nor log segments found. Creating new log.
I20260925 03:02:52.141223 23628 raft_consensus.cc:348] T 832e06098a3e499f82d53853e26235cf P 2edab657a33a4a0abfbd3262b1668483 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:02:52.141973 23628 raft_consensus.cc:374] T 832e06098a3e499f82d53853e26235cf P 2edab657a33a4a0abfbd3262b1668483 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:02:52.142236 23628 raft_consensus.cc:729] T 832e06098a3e499f82d53853e26235cf P 2edab657a33a4a0abfbd3262b1668483 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2edab657a33a4a0abfbd3262b1668483, State: Initialized, Role: FOLLOWER
I20260925 03:02:52.142827 23628 consensus_queue.cc:260] T 832e06098a3e499f82d53853e26235cf P 2edab657a33a4a0abfbd3262b1668483 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:02:52.145411 23628 ts_tablet_manager.cc:1428] T 832e06098a3e499f82d53853e26235cf P 2edab657a33a4a0abfbd3262b1668483: Time spent starting tablet: real 0.008s user 0.007s sys 0.001s
I20260925 03:02:52.146288 23628 tablet_bootstrap.cc:492] T 138dfa413a144a0e8911a2b402857cb4 P 2edab657a33a4a0abfbd3262b1668483: Bootstrap starting.
I20260925 03:02:52.152410 23628 tablet_bootstrap.cc:654] T 138dfa413a144a0e8911a2b402857cb4 P 2edab657a33a4a0abfbd3262b1668483: Neither blocks nor log segments found. Creating new log.
I20260925 03:02:52.154799 23627 tablet_bootstrap.cc:492] T 43de311d912b475eb01e1f7c9a65412a P b66c1d3fe3174f64b0936c512ee08f48: No bootstrap required, opened a new log
I20260925 03:02:52.155280 23627 ts_tablet_manager.cc:1397] T 43de311d912b475eb01e1f7c9a65412a P b66c1d3fe3174f64b0936c512ee08f48: Time spent bootstrapping tablet: real 0.036s user 0.007s sys 0.007s
I20260925 03:02:52.157938 23627 raft_consensus.cc:348] T 43de311d912b475eb01e1f7c9a65412a P b66c1d3fe3174f64b0936c512ee08f48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:02:52.158644 23627 raft_consensus.cc:374] T 43de311d912b475eb01e1f7c9a65412a P b66c1d3fe3174f64b0936c512ee08f48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:02:52.158921 23627 raft_consensus.cc:729] T 43de311d912b475eb01e1f7c9a65412a P b66c1d3fe3174f64b0936c512ee08f48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b66c1d3fe3174f64b0936c512ee08f48, State: Initialized, Role: FOLLOWER
I20260925 03:02:52.159516 23627 consensus_queue.cc:260] T 43de311d912b475eb01e1f7c9a65412a P b66c1d3fe3174f64b0936c512ee08f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:02:52.160502 23628 tablet_bootstrap.cc:492] T 138dfa413a144a0e8911a2b402857cb4 P 2edab657a33a4a0abfbd3262b1668483: No bootstrap required, opened a new log
I20260925 03:02:52.161645 23627 ts_tablet_manager.cc:1428] T 43de311d912b475eb01e1f7c9a65412a P b66c1d3fe3174f64b0936c512ee08f48: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260925 03:02:52.162518 23627 tablet_bootstrap.cc:492] T 832e06098a3e499f82d53853e26235cf P b66c1d3fe3174f64b0936c512ee08f48: Bootstrap starting.
I20260925 03:02:52.163563 23629 tablet_bootstrap.cc:492] T c30e9a1092fe4d99994480aab8863cf6 P fa551b23283b4034b256ba25acec541b: No bootstrap required, opened a new log
I20260925 03:02:52.164122 23629 ts_tablet_manager.cc:1397] T c30e9a1092fe4d99994480aab8863cf6 P fa551b23283b4034b256ba25acec541b: Time spent bootstrapping tablet: real 0.038s user 0.008s sys 0.008s
I20260925 03:02:52.166976 23628 ts_tablet_manager.cc:1397] T 138dfa413a144a0e8911a2b402857cb4 P 2edab657a33a4a0abfbd3262b1668483: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.008s
I20260925 03:02:52.168457 23627 tablet_bootstrap.cc:654] T 832e06098a3e499f82d53853e26235cf P b66c1d3fe3174f64b0936c512ee08f48: Neither blocks nor log segments found. Creating new log.
I20260925 03:02:52.169557 23628 raft_consensus.cc:348] T 138dfa413a144a0e8911a2b402857cb4 P 2edab657a33a4a0abfbd3262b1668483 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:02:52.170256 23628 raft_consensus.cc:374] T 138dfa413a144a0e8911a2b402857cb4 P 2edab657a33a4a0abfbd3262b1668483 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:02:52.170531 23628 raft_consensus.cc:729] T 138dfa413a144a0e8911a2b402857cb4 P 2edab657a33a4a0abfbd3262b1668483 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2edab657a33a4a0abfbd3262b1668483, State: Initialized, Role: FOLLOWER
I20260925 03:02:52.171149 23628 consensus_queue.cc:260] T 138dfa413a144a0e8911a2b402857cb4 P 2edab657a33a4a0abfbd3262b1668483 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:02:52.173808 23629 raft_consensus.cc:348] T c30e9a1092fe4d99994480aab8863cf6 P fa551b23283b4034b256ba25acec541b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:02:52.174505 23629 raft_consensus.cc:374] T c30e9a1092fe4d99994480aab8863cf6 P fa551b23283b4034b256ba25acec541b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:02:52.174770 23629 raft_consensus.cc:729] T c30e9a1092fe4d99994480aab8863cf6 P fa551b23283b4034b256ba25acec541b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa551b23283b4034b256ba25acec541b, State: Initialized, Role: FOLLOWER
I20260925 03:02:52.175387 23629 consensus_queue.cc:260] T c30e9a1092fe4d99994480aab8863cf6 P fa551b23283b4034b256ba25acec541b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:02:52.177875 23628 ts_tablet_manager.cc:1428] T 138dfa413a144a0e8911a2b402857cb4 P 2edab657a33a4a0abfbd3262b1668483: Time spent starting tablet: real 0.011s user 0.004s sys 0.002s
I20260925 03:02:52.179898 23629 ts_tablet_manager.cc:1428] T c30e9a1092fe4d99994480aab8863cf6 P fa551b23283b4034b256ba25acec541b: Time spent starting tablet: real 0.015s user 0.009s sys 0.001s
I20260925 03:02:52.180809 23629 tablet_bootstrap.cc:492] T 9de619025d19432c8df31a19bb6fc446 P fa551b23283b4034b256ba25acec541b: Bootstrap starting.
I20260925 03:02:52.188669 23629 tablet_bootstrap.cc:654] T 9de619025d19432c8df31a19bb6fc446 P fa551b23283b4034b256ba25acec541b: Neither blocks nor log segments found. Creating new log.
I20260925 03:02:52.196384 23627 tablet_bootstrap.cc:492] T 832e06098a3e499f82d53853e26235cf P b66c1d3fe3174f64b0936c512ee08f48: No bootstrap required, opened a new log
I20260925 03:02:52.196882 23627 ts_tablet_manager.cc:1397] T 832e06098a3e499f82d53853e26235cf P b66c1d3fe3174f64b0936c512ee08f48: Time spent bootstrapping tablet: real 0.035s user 0.016s sys 0.004s
I20260925 03:02:52.197523 23629 tablet_bootstrap.cc:492] T 9de619025d19432c8df31a19bb6fc446 P fa551b23283b4034b256ba25acec541b: No bootstrap required, opened a new log
I20260925 03:02:52.197957 23629 ts_tablet_manager.cc:1397] T 9de619025d19432c8df31a19bb6fc446 P fa551b23283b4034b256ba25acec541b: Time spent bootstrapping tablet: real 0.017s user 0.014s sys 0.000s
I20260925 03:02:52.199172 23583 raft_consensus.cc:3037] T 43de311d912b475eb01e1f7c9a65412a P b66c1d3fe3174f64b0936c512ee08f48 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:02:52.199167 23627 raft_consensus.cc:348] T 832e06098a3e499f82d53853e26235cf P b66c1d3fe3174f64b0936c512ee08f48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:02:52.200071 23627 raft_consensus.cc:374] T 832e06098a3e499f82d53853e26235cf P b66c1d3fe3174f64b0936c512ee08f48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:02:52.200340 23627 raft_consensus.cc:729] T 832e06098a3e499f82d53853e26235cf P b66c1d3fe3174f64b0936c512ee08f48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b66c1d3fe3174f64b0936c512ee08f48, State: Initialized, Role: FOLLOWER
I20260925 03:02:52.200959 23629 raft_consensus.cc:348] T 9de619025d19432c8df31a19bb6fc446 P fa551b23283b4034b256ba25acec541b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:02:52.201154 23627 consensus_queue.cc:260] T 832e06098a3e499f82d53853e26235cf P b66c1d3fe3174f64b0936c512ee08f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:02:52.201897 23629 raft_consensus.cc:374] T 9de619025d19432c8df31a19bb6fc446 P fa551b23283b4034b256ba25acec541b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:02:52.203576 23627 ts_tablet_manager.cc:1428] T 832e06098a3e499f82d53853e26235cf P b66c1d3fe3174f64b0936c512ee08f48: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260925 03:02:52.202220 23629 raft_consensus.cc:729] T 9de619025d19432c8df31a19bb6fc446 P fa551b23283b4034b256ba25acec541b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa551b23283b4034b256ba25acec541b, State: Initialized, Role: FOLLOWER
I20260925 03:02:52.205756 23629 consensus_queue.cc:260] T 9de619025d19432c8df31a19bb6fc446 P fa551b23283b4034b256ba25acec541b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:02:52.207729 23629 ts_tablet_manager.cc:1428] T 9de619025d19432c8df31a19bb6fc446 P fa551b23283b4034b256ba25acec541b: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260925 03:02:52.212944 23660 consensus_queue.cc:1035] T 43de311d912b475eb01e1f7c9a65412a P 2edab657a33a4a0abfbd3262b1668483 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 03:02:52.217626 23665 raft_consensus.cc:482] T fb663a8b564c494f98f56845f8a05d6c P fa551b23283b4034b256ba25acec541b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:02:52.218184 23665 raft_consensus.cc:504] T fb663a8b564c494f98f56845f8a05d6c P fa551b23283b4034b256ba25acec541b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:02:52.219723 23634 raft_consensus.cc:482] T c30e9a1092fe4d99994480aab8863cf6 P b66c1d3fe3174f64b0936c512ee08f48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:02:52.220247 23634 raft_consensus.cc:504] T c30e9a1092fe4d99994480aab8863cf6 P b66c1d3fe3174f64b0936c512ee08f48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:02:52.221596 23584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb663a8b564c494f98f56845f8a05d6c" candidate_uuid: "fa551b23283b4034b256ba25acec541b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b66c1d3fe3174f64b0936c512ee08f48" is_pre_election: true
I20260925 03:02:52.222231 23584 raft_consensus.cc:2454] T fb663a8b564c494f98f56845f8a05d6c P b66c1d3fe3174f64b0936c512ee08f48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa551b23283b4034b256ba25acec541b in term 0.
I20260925 03:02:52.222384 23634 leader_election.cc:288] T c30e9a1092fe4d99994480aab8863cf6 P b66c1d3fe3174f64b0936c512ee08f48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa551b23283b4034b256ba25acec541b (127.22.193.1:42181), 2edab657a33a4a0abfbd3262b1668483 (127.22.193.2:35431)
I20260925 03:02:52.223394 23398 leader_election.cc:302] T fb663a8b564c494f98f56845f8a05d6c P fa551b23283b4034b256ba25acec541b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b66c1d3fe3174f64b0936c512ee08f48, fa551b23283b4034b256ba25acec541b; no voters:
I20260925 03:02:52.224098 23435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c30e9a1092fe4d99994480aab8863cf6" candidate_uuid: "b66c1d3fe3174f64b0936c512ee08f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa551b23283b4034b256ba25acec541b" is_pre_election: true
I20260925 03:02:52.224886 23632 raft_consensus.cc:2781] T fb663a8b564c494f98f56845f8a05d6c P fa551b23283b4034b256ba25acec541b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 03:02:52.225255 23632 raft_consensus.cc:482] T fb663a8b564c494f98f56845f8a05d6c P fa551b23283b4034b256ba25acec541b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:02:52.225564 23632 raft_consensus.cc:3037] T fb663a8b564c494f98f56845f8a05d6c P fa551b23283b4034b256ba25acec541b [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:02:52.225468 23510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb663a8b564c494f98f56845f8a05d6c" candidate_uuid: "fa551b23283b4034b256ba25acec541b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2edab657a33a4a0abfbd3262b1668483" is_pre_election: true
I20260925 03:02:52.226150 23510 raft_consensus.cc:2454] T fb663a8b564c494f98f56845f8a05d6c P 2edab657a33a4a0abfbd3262b1668483 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa551b23283b4034b256ba25acec541b in term 0.
I20260925 03:02:52.220220 23665 leader_election.cc:288] T fb663a8b564c494f98f56845f8a05d6c P fa551b23283b4034b256ba25acec541b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2edab657a33a4a0abfbd3262b1668483 (127.22.193.2:35431), b66c1d3fe3174f64b0936c512ee08f48 (127.22.193.3:42621)
I20260925 03:02:52.229836 23435 raft_consensus.cc:2454] T c30e9a1092fe4d99994480aab8863cf6 P fa551b23283b4034b256ba25acec541b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b66c1d3fe3174f64b0936c512ee08f48 in term 0.
I20260925 03:02:52.230850 23510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c30e9a1092fe4d99994480aab8863cf6" candidate_uuid: "b66c1d3fe3174f64b0936c512ee08f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2edab657a33a4a0abfbd3262b1668483" is_pre_election: true
I20260925 03:02:52.231474 23510 raft_consensus.cc:2454] T c30e9a1092fe4d99994480aab8863cf6 P 2edab657a33a4a0abfbd3262b1668483 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b66c1d3fe3174f64b0936c512ee08f48 in term 0.
I20260925 03:02:52.233121 23547 leader_election.cc:302] T c30e9a1092fe4d99994480aab8863cf6 P b66c1d3fe3174f64b0936c512ee08f48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b66c1d3fe3174f64b0936c512ee08f48, fa551b23283b4034b256ba25acec541b; no voters:
I20260925 03:02:52.234798 23634 raft_consensus.cc:2781] T c30e9a1092fe4d99994480aab8863cf6 P b66c1d3fe3174f64b0936c512ee08f48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 03:02:52.236078 23634 raft_consensus.cc:482] T c30e9a1092fe4d99994480aab8863cf6 P b66c1d3fe3174f64b0936c512ee08f48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:02:52.236410 23634 raft_consensus.cc:3037] T c30e9a1092fe4d99994480aab8863cf6 P b66c1d3fe3174f64b0936c512ee08f48 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:02:52.243929 23632 raft_consensus.cc:504] T fb663a8b564c494f98f56845f8a05d6c P fa551b23283b4034b256ba25acec541b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:02:52.248093 23510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb663a8b564c494f98f56845f8a05d6c" candidate_uuid: "fa551b23283b4034b256ba25acec541b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2edab657a33a4a0abfbd3262b1668483"
I20260925 03:02:52.248821 23510 raft_consensus.cc:3037] T fb663a8b564c494f98f56845f8a05d6c P 2edab657a33a4a0abfbd3262b1668483 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:02:52.248140 23584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb663a8b564c494f98f56845f8a05d6c" candidate_uuid: "fa551b23283b4034b256ba25acec541b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b66c1d3fe3174f64b0936c512ee08f48"
I20260925 03:02:52.254763 23584 raft_consensus.cc:3037] T fb663a8b564c494f98f56845f8a05d6c P b66c1d3fe3174f64b0936c512ee08f48 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:02:52.258878 23635 consensus_queue.cc:1035] T 43de311d912b475eb01e1f7c9a65412a P 2edab657a33a4a0abfbd3262b1668483 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 03:02:52.262255 23634 raft_consensus.cc:504] T c30e9a1092fe4d99994480aab8863cf6 P b66c1d3fe3174f64b0936c512ee08f48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:02:52.266438 23634 leader_election.cc:288] T c30e9a1092fe4d99994480aab8863cf6 P b66c1d3fe3174f64b0936c512ee08f48 [CANDIDATE]: Term 1 election: Requested vote from peers fa551b23283b4034b256ba25acec541b (127.22.193.1:42181), 2edab657a33a4a0abfbd3262b1668483 (127.22.193.2:35431)
I20260925 03:02:52.266536 23434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c30e9a1092fe4d99994480aab8863cf6" candidate_uuid: "b66c1d3fe3174f64b0936c512ee08f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa551b23283b4034b256ba25acec541b"
I20260925 03:02:52.267278 23434 raft_consensus.cc:3037] T c30e9a1092fe4d99994480aab8863cf6 P fa551b23283b4034b256ba25acec541b [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:02:52.269958 23584 raft_consensus.cc:2454] T fb663a8b564c494f98f56845f8a05d6c P b66c1d3fe3174f64b0936c512ee08f48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa551b23283b4034b256ba25acec541b in term 1.
I20260925 03:02:52.248293 23632 leader_election.cc:288] T fb663a8b564c494f98f56845f8a05d6c P fa551b23283b4034b256ba25acec541b [CANDIDATE]: Term 1 election: Requested vote from peers 2edab657a33a4a0abfbd3262b1668483 (127.22.193.2:35431), b66c1d3fe3174f64b0936c512ee08f48 (127.22.193.3:42621)
I20260925 03:02:52.271926 23398 leader_election.cc:302] T fb663a8b564c494f98f56845f8a05d6c P fa551b23283b4034b256ba25acec541b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b66c1d3fe3174f64b0936c512ee08f48, fa551b23283b4034b256ba25acec541b; no voters:
I20260925 03:02:52.272404 23510 raft_consensus.cc:2454] T fb663a8b564c494f98f56845f8a05d6c P 2edab657a33a4a0abfbd3262b1668483 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa551b23283b4034b256ba25acec541b in term 1.
I20260925 03:02:52.275101 23632 raft_consensus.cc:2781] T fb663a8b564c494f98f56845f8a05d6c P fa551b23283b4034b256ba25acec541b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:02:52.275329 23509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c30e9a1092fe4d99994480aab8863cf6" candidate_uuid: "b66c1d3fe3174f64b0936c512ee08f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2edab657a33a4a0abfbd3262b1668483"
I20260925 03:02:52.275969 23509 raft_consensus.cc:3037] T c30e9a1092fe4d99994480aab8863cf6 P 2edab657a33a4a0abfbd3262b1668483 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:02:52.276322 23632 raft_consensus.cc:686] T fb663a8b564c494f98f56845f8a05d6c P fa551b23283b4034b256ba25acec541b [term 1 LEADER]: Becoming Leader. State: Replica: fa551b23283b4034b256ba25acec541b, State: Running, Role: LEADER
I20260925 03:02:52.277956 23632 consensus_queue.cc:237] T fb663a8b564c494f98f56845f8a05d6c P fa551b23283b4034b256ba25acec541b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:02:52.279985 23665 raft_consensus.cc:482] T 9de619025d19432c8df31a19bb6fc446 P fa551b23283b4034b256ba25acec541b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:02:52.280445 23665 raft_consensus.cc:504] T 9de619025d19432c8df31a19bb6fc446 P fa551b23283b4034b256ba25acec541b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:02:52.282552 23665 leader_election.cc:288] T 9de619025d19432c8df31a19bb6fc446 P fa551b23283b4034b256ba25acec541b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2edab657a33a4a0abfbd3262b1668483 (127.22.193.2:35431), b66c1d3fe3174f64b0936c512ee08f48 (127.22.193.3:42621)
I20260925 03:02:52.284747 23510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9de619025d19432c8df31a19bb6fc446" candidate_uuid: "fa551b23283b4034b256ba25acec541b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2edab657a33a4a0abfbd3262b1668483" is_pre_election: true
I20260925 03:02:52.284970 23434 raft_consensus.cc:2454] T c30e9a1092fe4d99994480aab8863cf6 P fa551b23283b4034b256ba25acec541b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b66c1d3fe3174f64b0936c512ee08f48 in term 1.
I20260925 03:02:52.285434 23510 raft_consensus.cc:2454] T 9de619025d19432c8df31a19bb6fc446 P 2edab657a33a4a0abfbd3262b1668483 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa551b23283b4034b256ba25acec541b in term 0.
I20260925 03:02:52.286046 23547 leader_election.cc:302] T c30e9a1092fe4d99994480aab8863cf6 P b66c1d3fe3174f64b0936c512ee08f48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b66c1d3fe3174f64b0936c512ee08f48, fa551b23283b4034b256ba25acec541b; no voters:
I20260925 03:02:52.288285 23682 raft_consensus.cc:2781] T c30e9a1092fe4d99994480aab8863cf6 P b66c1d3fe3174f64b0936c512ee08f48 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:02:52.288928 23584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9de619025d19432c8df31a19bb6fc446" candidate_uuid: "fa551b23283b4034b256ba25acec541b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b66c1d3fe3174f64b0936c512ee08f48" is_pre_election: true
I20260925 03:02:52.289933 23584 raft_consensus.cc:2454] T 9de619025d19432c8df31a19bb6fc446 P b66c1d3fe3174f64b0936c512ee08f48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa551b23283b4034b256ba25acec541b in term 0.
I20260925 03:02:52.291824 23399 leader_election.cc:302] T 9de619025d19432c8df31a19bb6fc446 P fa551b23283b4034b256ba25acec541b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2edab657a33a4a0abfbd3262b1668483, fa551b23283b4034b256ba25acec541b; no voters:
I20260925 03:02:52.293428 23665 raft_consensus.cc:2781] T 9de619025d19432c8df31a19bb6fc446 P fa551b23283b4034b256ba25acec541b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 03:02:52.293768 23665 raft_consensus.cc:482] T 9de619025d19432c8df31a19bb6fc446 P fa551b23283b4034b256ba25acec541b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:02:52.294082 23665 raft_consensus.cc:3037] T 9de619025d19432c8df31a19bb6fc446 P fa551b23283b4034b256ba25acec541b [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:02:52.297288 23682 raft_consensus.cc:686] T c30e9a1092fe4d99994480aab8863cf6 P b66c1d3fe3174f64b0936c512ee08f48 [term 1 LEADER]: Becoming Leader. State: Replica: b66c1d3fe3174f64b0936c512ee08f48, State: Running, Role: LEADER
I20260925 03:02:52.298194 23682 consensus_queue.cc:237] T c30e9a1092fe4d99994480aab8863cf6 P b66c1d3fe3174f64b0936c512ee08f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:02:52.304632 23509 raft_consensus.cc:2454] T c30e9a1092fe4d99994480aab8863cf6 P 2edab657a33a4a0abfbd3262b1668483 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b66c1d3fe3174f64b0936c512ee08f48 in term 1.
I20260925 03:02:52.307005 23665 raft_consensus.cc:504] T 9de619025d19432c8df31a19bb6fc446 P fa551b23283b4034b256ba25acec541b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:02:52.310750 23665 leader_election.cc:288] T 9de619025d19432c8df31a19bb6fc446 P fa551b23283b4034b256ba25acec541b [CANDIDATE]: Term 1 election: Requested vote from peers 2edab657a33a4a0abfbd3262b1668483 (127.22.193.2:35431), b66c1d3fe3174f64b0936c512ee08f48 (127.22.193.3:42621)
I20260925 03:02:52.311931 23509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9de619025d19432c8df31a19bb6fc446" candidate_uuid: "fa551b23283b4034b256ba25acec541b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2edab657a33a4a0abfbd3262b1668483"
I20260925 03:02:52.312085 23583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9de619025d19432c8df31a19bb6fc446" candidate_uuid: "fa551b23283b4034b256ba25acec541b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b66c1d3fe3174f64b0936c512ee08f48"
I20260925 03:02:52.312592 23509 raft_consensus.cc:3037] T 9de619025d19432c8df31a19bb6fc446 P 2edab657a33a4a0abfbd3262b1668483 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:02:52.312850 23583 raft_consensus.cc:3037] T 9de619025d19432c8df31a19bb6fc446 P b66c1d3fe3174f64b0936c512ee08f48 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:02:52.324224 23583 raft_consensus.cc:2454] T 9de619025d19432c8df31a19bb6fc446 P b66c1d3fe3174f64b0936c512ee08f48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa551b23283b4034b256ba25acec541b in term 1.
I20260925 03:02:52.326829 23398 leader_election.cc:302] T 9de619025d19432c8df31a19bb6fc446 P fa551b23283b4034b256ba25acec541b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b66c1d3fe3174f64b0936c512ee08f48, fa551b23283b4034b256ba25acec541b; no voters:
I20260925 03:02:52.328495 23686 raft_consensus.cc:2781] T 9de619025d19432c8df31a19bb6fc446 P fa551b23283b4034b256ba25acec541b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:02:52.329047 23686 raft_consensus.cc:686] T 9de619025d19432c8df31a19bb6fc446 P fa551b23283b4034b256ba25acec541b [term 1 LEADER]: Becoming Leader. State: Replica: fa551b23283b4034b256ba25acec541b, State: Running, Role: LEADER
I20260925 03:02:52.329921 23686 consensus_queue.cc:237] T 9de619025d19432c8df31a19bb6fc446 P fa551b23283b4034b256ba25acec541b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:02:52.332755 23509 raft_consensus.cc:2454] T 9de619025d19432c8df31a19bb6fc446 P 2edab657a33a4a0abfbd3262b1668483 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa551b23283b4034b256ba25acec541b in term 1.
I20260925 03:02:52.333900 23330 catalog_manager.cc:5495] T c30e9a1092fe4d99994480aab8863cf6 P b66c1d3fe3174f64b0936c512ee08f48 reported cstate change: term changed from 0 to 1, leader changed from <none> to b66c1d3fe3174f64b0936c512ee08f48 (127.22.193.3). New cstate: current_term: 1 leader_uuid: "b66c1d3fe3174f64b0936c512ee08f48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } health_report { overall_health: HEALTHY } } }
I20260925 03:02:52.343181 23329 catalog_manager.cc:5495] T fb663a8b564c494f98f56845f8a05d6c P fa551b23283b4034b256ba25acec541b reported cstate change: term changed from 0 to 1, leader changed from <none> to fa551b23283b4034b256ba25acec541b (127.22.193.1). New cstate: current_term: 1 leader_uuid: "fa551b23283b4034b256ba25acec541b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } health_report { overall_health: UNKNOWN } } }
I20260925 03:02:52.345130 23329 catalog_manager.cc:5495] T 9de619025d19432c8df31a19bb6fc446 P fa551b23283b4034b256ba25acec541b reported cstate change: term changed from 0 to 1, leader changed from <none> to fa551b23283b4034b256ba25acec541b (127.22.193.1). New cstate: current_term: 1 leader_uuid: "fa551b23283b4034b256ba25acec541b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } health_report { overall_health: UNKNOWN } } }
I20260925 03:02:52.414710 23682 raft_consensus.cc:482] T 138dfa413a144a0e8911a2b402857cb4 P b66c1d3fe3174f64b0936c512ee08f48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:02:52.415463 23682 raft_consensus.cc:504] T 138dfa413a144a0e8911a2b402857cb4 P b66c1d3fe3174f64b0936c512ee08f48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:02:52.417603 23682 leader_election.cc:288] T 138dfa413a144a0e8911a2b402857cb4 P b66c1d3fe3174f64b0936c512ee08f48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa551b23283b4034b256ba25acec541b (127.22.193.1:42181), 2edab657a33a4a0abfbd3262b1668483 (127.22.193.2:35431)
I20260925 03:02:52.418552 23509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "138dfa413a144a0e8911a2b402857cb4" candidate_uuid: "b66c1d3fe3174f64b0936c512ee08f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2edab657a33a4a0abfbd3262b1668483" is_pre_election: true
I20260925 03:02:52.418558 23435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "138dfa413a144a0e8911a2b402857cb4" candidate_uuid: "b66c1d3fe3174f64b0936c512ee08f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa551b23283b4034b256ba25acec541b" is_pre_election: true
I20260925 03:02:52.419363 23509 raft_consensus.cc:2454] T 138dfa413a144a0e8911a2b402857cb4 P 2edab657a33a4a0abfbd3262b1668483 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b66c1d3fe3174f64b0936c512ee08f48 in term 0.
I20260925 03:02:52.419414 23435 raft_consensus.cc:2454] T 138dfa413a144a0e8911a2b402857cb4 P fa551b23283b4034b256ba25acec541b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b66c1d3fe3174f64b0936c512ee08f48 in term 0.
I20260925 03:02:52.420284 23548 leader_election.cc:302] T 138dfa413a144a0e8911a2b402857cb4 P b66c1d3fe3174f64b0936c512ee08f48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2edab657a33a4a0abfbd3262b1668483, b66c1d3fe3174f64b0936c512ee08f48; no voters:
I20260925 03:02:52.421075 23682 raft_consensus.cc:2781] T 138dfa413a144a0e8911a2b402857cb4 P b66c1d3fe3174f64b0936c512ee08f48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 03:02:52.421439 23682 raft_consensus.cc:482] T 138dfa413a144a0e8911a2b402857cb4 P b66c1d3fe3174f64b0936c512ee08f48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:02:52.421763 23682 raft_consensus.cc:3037] T 138dfa413a144a0e8911a2b402857cb4 P b66c1d3fe3174f64b0936c512ee08f48 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:02:52.430258 23682 raft_consensus.cc:504] T 138dfa413a144a0e8911a2b402857cb4 P b66c1d3fe3174f64b0936c512ee08f48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:02:52.433466 23682 leader_election.cc:288] T 138dfa413a144a0e8911a2b402857cb4 P b66c1d3fe3174f64b0936c512ee08f48 [CANDIDATE]: Term 1 election: Requested vote from peers fa551b23283b4034b256ba25acec541b (127.22.193.1:42181), 2edab657a33a4a0abfbd3262b1668483 (127.22.193.2:35431)
I20260925 03:02:52.433848 23435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "138dfa413a144a0e8911a2b402857cb4" candidate_uuid: "b66c1d3fe3174f64b0936c512ee08f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa551b23283b4034b256ba25acec541b"
I20260925 03:02:52.434525 23435 raft_consensus.cc:3037] T 138dfa413a144a0e8911a2b402857cb4 P fa551b23283b4034b256ba25acec541b [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:02:52.435642 23509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "138dfa413a144a0e8911a2b402857cb4" candidate_uuid: "b66c1d3fe3174f64b0936c512ee08f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2edab657a33a4a0abfbd3262b1668483"
I20260925 03:02:52.436298 23509 raft_consensus.cc:3037] T 138dfa413a144a0e8911a2b402857cb4 P 2edab657a33a4a0abfbd3262b1668483 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:02:52.441272 23435 raft_consensus.cc:2454] T 138dfa413a144a0e8911a2b402857cb4 P fa551b23283b4034b256ba25acec541b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b66c1d3fe3174f64b0936c512ee08f48 in term 1.
I20260925 03:02:52.442951 23547 leader_election.cc:302] T 138dfa413a144a0e8911a2b402857cb4 P b66c1d3fe3174f64b0936c512ee08f48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b66c1d3fe3174f64b0936c512ee08f48, fa551b23283b4034b256ba25acec541b; no voters:
I20260925 03:02:52.444052 23509 raft_consensus.cc:2454] T 138dfa413a144a0e8911a2b402857cb4 P 2edab657a33a4a0abfbd3262b1668483 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b66c1d3fe3174f64b0936c512ee08f48 in term 1.
I20260925 03:02:52.445227 23682 raft_consensus.cc:2781] T 138dfa413a144a0e8911a2b402857cb4 P b66c1d3fe3174f64b0936c512ee08f48 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:02:52.445744 23682 raft_consensus.cc:686] T 138dfa413a144a0e8911a2b402857cb4 P b66c1d3fe3174f64b0936c512ee08f48 [term 1 LEADER]: Becoming Leader. State: Replica: b66c1d3fe3174f64b0936c512ee08f48, State: Running, Role: LEADER
I20260925 03:02:52.446509 23682 consensus_queue.cc:237] T 138dfa413a144a0e8911a2b402857cb4 P b66c1d3fe3174f64b0936c512ee08f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:02:52.450115 23635 consensus_queue.cc:1035] T ef74cafe53e34e1dace8b02d73693521 P 2edab657a33a4a0abfbd3262b1668483 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 03:02:52.458710 23329 catalog_manager.cc:5495] T 138dfa413a144a0e8911a2b402857cb4 P b66c1d3fe3174f64b0936c512ee08f48 reported cstate change: term changed from 0 to 1, leader changed from <none> to b66c1d3fe3174f64b0936c512ee08f48 (127.22.193.3). New cstate: current_term: 1 leader_uuid: "b66c1d3fe3174f64b0936c512ee08f48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } health_report { overall_health: HEALTHY } } }
I20260925 03:02:52.476202 23635 consensus_queue.cc:1035] T ef74cafe53e34e1dace8b02d73693521 P 2edab657a33a4a0abfbd3262b1668483 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 03:02:52.526822 23682 raft_consensus.cc:482] T 832e06098a3e499f82d53853e26235cf P b66c1d3fe3174f64b0936c512ee08f48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:02:52.527318 23682 raft_consensus.cc:504] T 832e06098a3e499f82d53853e26235cf P b66c1d3fe3174f64b0936c512ee08f48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:02:52.528970 23682 leader_election.cc:288] T 832e06098a3e499f82d53853e26235cf P b66c1d3fe3174f64b0936c512ee08f48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa551b23283b4034b256ba25acec541b (127.22.193.1:42181), 2edab657a33a4a0abfbd3262b1668483 (127.22.193.2:35431)
I20260925 03:02:52.529950 23435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "832e06098a3e499f82d53853e26235cf" candidate_uuid: "b66c1d3fe3174f64b0936c512ee08f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa551b23283b4034b256ba25acec541b" is_pre_election: true
I20260925 03:02:52.530025 23509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "832e06098a3e499f82d53853e26235cf" candidate_uuid: "b66c1d3fe3174f64b0936c512ee08f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2edab657a33a4a0abfbd3262b1668483" is_pre_election: true
I20260925 03:02:52.530658 23435 raft_consensus.cc:2454] T 832e06098a3e499f82d53853e26235cf P fa551b23283b4034b256ba25acec541b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b66c1d3fe3174f64b0936c512ee08f48 in term 0.
I20260925 03:02:52.530767 23509 raft_consensus.cc:2454] T 832e06098a3e499f82d53853e26235cf P 2edab657a33a4a0abfbd3262b1668483 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b66c1d3fe3174f64b0936c512ee08f48 in term 0.
I20260925 03:02:52.531744 23547 leader_election.cc:302] T 832e06098a3e499f82d53853e26235cf P b66c1d3fe3174f64b0936c512ee08f48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b66c1d3fe3174f64b0936c512ee08f48, fa551b23283b4034b256ba25acec541b; no voters:
I20260925 03:02:52.532560 23682 raft_consensus.cc:2781] T 832e06098a3e499f82d53853e26235cf P b66c1d3fe3174f64b0936c512ee08f48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 03:02:52.532955 23682 raft_consensus.cc:482] T 832e06098a3e499f82d53853e26235cf P b66c1d3fe3174f64b0936c512ee08f48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:02:52.533210 23682 raft_consensus.cc:3037] T 832e06098a3e499f82d53853e26235cf P b66c1d3fe3174f64b0936c512ee08f48 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:02:52.538005 23682 raft_consensus.cc:504] T 832e06098a3e499f82d53853e26235cf P b66c1d3fe3174f64b0936c512ee08f48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:02:52.539523 23682 leader_election.cc:288] T 832e06098a3e499f82d53853e26235cf P b66c1d3fe3174f64b0936c512ee08f48 [CANDIDATE]: Term 1 election: Requested vote from peers fa551b23283b4034b256ba25acec541b (127.22.193.1:42181), 2edab657a33a4a0abfbd3262b1668483 (127.22.193.2:35431)
I20260925 03:02:52.540503 23435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "832e06098a3e499f82d53853e26235cf" candidate_uuid: "b66c1d3fe3174f64b0936c512ee08f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa551b23283b4034b256ba25acec541b"
I20260925 03:02:52.540652 23509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "832e06098a3e499f82d53853e26235cf" candidate_uuid: "b66c1d3fe3174f64b0936c512ee08f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2edab657a33a4a0abfbd3262b1668483"
I20260925 03:02:52.541085 23435 raft_consensus.cc:3037] T 832e06098a3e499f82d53853e26235cf P fa551b23283b4034b256ba25acec541b [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:02:52.541332 23509 raft_consensus.cc:3037] T 832e06098a3e499f82d53853e26235cf P 2edab657a33a4a0abfbd3262b1668483 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:02:52.546504 23435 raft_consensus.cc:2454] T 832e06098a3e499f82d53853e26235cf P fa551b23283b4034b256ba25acec541b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b66c1d3fe3174f64b0936c512ee08f48 in term 1.
I20260925 03:02:52.547591 23547 leader_election.cc:302] T 832e06098a3e499f82d53853e26235cf P b66c1d3fe3174f64b0936c512ee08f48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b66c1d3fe3174f64b0936c512ee08f48, fa551b23283b4034b256ba25acec541b; no voters:
I20260925 03:02:52.548326 23682 raft_consensus.cc:2781] T 832e06098a3e499f82d53853e26235cf P b66c1d3fe3174f64b0936c512ee08f48 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:02:52.548768 23682 raft_consensus.cc:686] T 832e06098a3e499f82d53853e26235cf P b66c1d3fe3174f64b0936c512ee08f48 [term 1 LEADER]: Becoming Leader. State: Replica: b66c1d3fe3174f64b0936c512ee08f48, State: Running, Role: LEADER
I20260925 03:02:52.548770 23509 raft_consensus.cc:2454] T 832e06098a3e499f82d53853e26235cf P 2edab657a33a4a0abfbd3262b1668483 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b66c1d3fe3174f64b0936c512ee08f48 in term 1.
I20260925 03:02:52.549921 23682 consensus_queue.cc:237] T 832e06098a3e499f82d53853e26235cf P b66c1d3fe3174f64b0936c512ee08f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:02:52.561774 23332 catalog_manager.cc:5495] T 832e06098a3e499f82d53853e26235cf P b66c1d3fe3174f64b0936c512ee08f48 reported cstate change: term changed from 0 to 1, leader changed from <none> to b66c1d3fe3174f64b0936c512ee08f48 (127.22.193.3). New cstate: current_term: 1 leader_uuid: "b66c1d3fe3174f64b0936c512ee08f48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } health_report { overall_health: HEALTHY } } }
I20260925 03:02:52.564348 23635 consensus_queue.cc:1035] T 2baf62cd63564f4ba08b201e40e19d55 P 2edab657a33a4a0abfbd3262b1668483 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260925 03:02:52.569154 23637 consensus_queue.cc:1035] T 2baf62cd63564f4ba08b201e40e19d55 P 2edab657a33a4a0abfbd3262b1668483 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260925 03:02:52.618026 23660 consensus_queue.cc:1035] T d01220d66a8c4d34ac86fc3559fa5165 P 2edab657a33a4a0abfbd3262b1668483 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 03:02:52.619415 23660 consensus_queue.cc:1035] T d01220d66a8c4d34ac86fc3559fa5165 P 2edab657a33a4a0abfbd3262b1668483 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 03:02:52.655423 23509 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ef74cafe53e34e1dace8b02d73693521"
dest_uuid: "2edab657a33a4a0abfbd3262b1668483"
mode: GRACEFUL
new_leader_uuid: "fa551b23283b4034b256ba25acec541b"
from {username='slave'} at 127.0.0.1:38976
I20260925 03:02:52.656461 23509 raft_consensus.cc:595] T ef74cafe53e34e1dace8b02d73693521 P 2edab657a33a4a0abfbd3262b1668483 [term 1 LEADER]: Received request to transfer leadership to TS fa551b23283b4034b256ba25acec541b
I20260925 03:02:52.663072 23509 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d01220d66a8c4d34ac86fc3559fa5165"
dest_uuid: "2edab657a33a4a0abfbd3262b1668483"
mode: GRACEFUL
new_leader_uuid: "fa551b23283b4034b256ba25acec541b"
from {username='slave'} at 127.0.0.1:38976
I20260925 03:02:52.663730 23509 raft_consensus.cc:595] T d01220d66a8c4d34ac86fc3559fa5165 P 2edab657a33a4a0abfbd3262b1668483 [term 1 LEADER]: Received request to transfer leadership to TS fa551b23283b4034b256ba25acec541b
I20260925 03:02:52.679605 23583 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "832e06098a3e499f82d53853e26235cf"
dest_uuid: "b66c1d3fe3174f64b0936c512ee08f48"
mode: GRACEFUL
new_leader_uuid: "fa551b23283b4034b256ba25acec541b"
from {username='slave'} at 127.0.0.1:51192
I20260925 03:02:52.680397 23583 raft_consensus.cc:595] T 832e06098a3e499f82d53853e26235cf P b66c1d3fe3174f64b0936c512ee08f48 [term 1 LEADER]: Received request to transfer leadership to TS fa551b23283b4034b256ba25acec541b
I20260925 03:02:52.685717 23509 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2baf62cd63564f4ba08b201e40e19d55"
dest_uuid: "2edab657a33a4a0abfbd3262b1668483"
mode: GRACEFUL
new_leader_uuid: "fa551b23283b4034b256ba25acec541b"
from {username='slave'} at 127.0.0.1:38976
I20260925 03:02:52.686250 23509 raft_consensus.cc:595] T 2baf62cd63564f4ba08b201e40e19d55 P 2edab657a33a4a0abfbd3262b1668483 [term 1 LEADER]: Received request to transfer leadership to TS fa551b23283b4034b256ba25acec541b
I20260925 03:02:52.692384 23583 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c30e9a1092fe4d99994480aab8863cf6"
dest_uuid: "b66c1d3fe3174f64b0936c512ee08f48"
mode: GRACEFUL
new_leader_uuid: "2edab657a33a4a0abfbd3262b1668483"
from {username='slave'} at 127.0.0.1:51192
I20260925 03:02:52.693296 23583 raft_consensus.cc:595] T c30e9a1092fe4d99994480aab8863cf6 P b66c1d3fe3174f64b0936c512ee08f48 [term 1 LEADER]: Received request to transfer leadership to TS 2edab657a33a4a0abfbd3262b1668483
I20260925 03:02:52.703644 23682 consensus_queue.cc:1035] T c30e9a1092fe4d99994480aab8863cf6 P b66c1d3fe3174f64b0936c512ee08f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260925 03:02:52.712888 23634 raft_consensus.cc:979] T c30e9a1092fe4d99994480aab8863cf6 P b66c1d3fe3174f64b0936c512ee08f48: : Instructing follower 2edab657a33a4a0abfbd3262b1668483 to start an election
I20260925 03:02:52.714711 23682 raft_consensus.cc:1067] T c30e9a1092fe4d99994480aab8863cf6 P b66c1d3fe3174f64b0936c512ee08f48 [term 1 LEADER]: Signalling peer 2edab657a33a4a0abfbd3262b1668483 to start an election
I20260925 03:02:52.714924 23435 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fb663a8b564c494f98f56845f8a05d6c"
dest_uuid: "fa551b23283b4034b256ba25acec541b"
mode: GRACEFUL
new_leader_uuid: "2edab657a33a4a0abfbd3262b1668483"
from {username='slave'} at 127.0.0.1:40566
I20260925 03:02:52.715523 23435 raft_consensus.cc:595] T fb663a8b564c494f98f56845f8a05d6c P fa551b23283b4034b256ba25acec541b [term 1 LEADER]: Received request to transfer leadership to TS 2edab657a33a4a0abfbd3262b1668483
I20260925 03:02:52.717363 23508 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c30e9a1092fe4d99994480aab8863cf6"
dest_uuid: "2edab657a33a4a0abfbd3262b1668483"
from {username='slave'} at 127.0.0.1:38970
I20260925 03:02:52.718443 23508 raft_consensus.cc:482] T c30e9a1092fe4d99994480aab8863cf6 P 2edab657a33a4a0abfbd3262b1668483 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 03:02:52.718814 23508 raft_consensus.cc:3037] T c30e9a1092fe4d99994480aab8863cf6 P 2edab657a33a4a0abfbd3262b1668483 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:02:52.724506 23508 raft_consensus.cc:504] T c30e9a1092fe4d99994480aab8863cf6 P 2edab657a33a4a0abfbd3262b1668483 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:02:52.727437 23435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c30e9a1092fe4d99994480aab8863cf6" candidate_uuid: "2edab657a33a4a0abfbd3262b1668483" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "fa551b23283b4034b256ba25acec541b"
I20260925 03:02:52.728124 23435 raft_consensus.cc:3037] T c30e9a1092fe4d99994480aab8863cf6 P fa551b23283b4034b256ba25acec541b [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:02:52.735596 23508 leader_election.cc:288] T c30e9a1092fe4d99994480aab8863cf6 P 2edab657a33a4a0abfbd3262b1668483 [CANDIDATE]: Term 2 election: Requested vote from peers fa551b23283b4034b256ba25acec541b (127.22.193.1:42181), b66c1d3fe3174f64b0936c512ee08f48 (127.22.193.3:42621)
I20260925 03:02:52.737592 23583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c30e9a1092fe4d99994480aab8863cf6" candidate_uuid: "2edab657a33a4a0abfbd3262b1668483" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b66c1d3fe3174f64b0936c512ee08f48"
I20260925 03:02:52.738240 23583 raft_consensus.cc:3032] T c30e9a1092fe4d99994480aab8863cf6 P b66c1d3fe3174f64b0936c512ee08f48 [term 1 LEADER]: Stepping down as leader of term 1
I20260925 03:02:52.738588 23583 raft_consensus.cc:729] T c30e9a1092fe4d99994480aab8863cf6 P b66c1d3fe3174f64b0936c512ee08f48 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b66c1d3fe3174f64b0936c512ee08f48, State: Running, Role: LEADER
I20260925 03:02:52.739375 23583 consensus_queue.cc:260] T c30e9a1092fe4d99994480aab8863cf6 P b66c1d3fe3174f64b0936c512ee08f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:02:52.740619 23583 raft_consensus.cc:3037] T c30e9a1092fe4d99994480aab8863cf6 P b66c1d3fe3174f64b0936c512ee08f48 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:02:52.742216 23584 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "138dfa413a144a0e8911a2b402857cb4"
dest_uuid: "b66c1d3fe3174f64b0936c512ee08f48"
mode: GRACEFUL
new_leader_uuid: "2edab657a33a4a0abfbd3262b1668483"
from {username='slave'} at 127.0.0.1:51192
I20260925 03:02:52.742426 23435 raft_consensus.cc:2454] T c30e9a1092fe4d99994480aab8863cf6 P fa551b23283b4034b256ba25acec541b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2edab657a33a4a0abfbd3262b1668483 in term 2.
I20260925 03:02:52.742830 23584 raft_consensus.cc:595] T 138dfa413a144a0e8911a2b402857cb4 P b66c1d3fe3174f64b0936c512ee08f48 [term 1 LEADER]: Received request to transfer leadership to TS 2edab657a33a4a0abfbd3262b1668483
I20260925 03:02:52.743072 23434 raft_consensus.cc:1226] T c30e9a1092fe4d99994480aab8863cf6 P fa551b23283b4034b256ba25acec541b [term 2 FOLLOWER]: Rejecting Update request from peer b66c1d3fe3174f64b0936c512ee08f48 for earlier term 1. Current term is 2. Ops: []
I20260925 03:02:52.748070 23583 raft_consensus.cc:2454] T c30e9a1092fe4d99994480aab8863cf6 P b66c1d3fe3174f64b0936c512ee08f48 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2edab657a33a4a0abfbd3262b1668483 in term 2.
I20260925 03:02:52.748042 23473 leader_election.cc:302] T c30e9a1092fe4d99994480aab8863cf6 P 2edab657a33a4a0abfbd3262b1668483 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2edab657a33a4a0abfbd3262b1668483, fa551b23283b4034b256ba25acec541b; no voters:
I20260925 03:02:52.749246 23695 raft_consensus.cc:2781] T c30e9a1092fe4d99994480aab8863cf6 P 2edab657a33a4a0abfbd3262b1668483 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 03:02:52.749720 23695 raft_consensus.cc:686] T c30e9a1092fe4d99994480aab8863cf6 P 2edab657a33a4a0abfbd3262b1668483 [term 2 LEADER]: Becoming Leader. State: Replica: 2edab657a33a4a0abfbd3262b1668483, State: Running, Role: LEADER
I20260925 03:02:52.749784 23435 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9de619025d19432c8df31a19bb6fc446"
dest_uuid: "fa551b23283b4034b256ba25acec541b"
mode: GRACEFUL
new_leader_uuid: "b66c1d3fe3174f64b0936c512ee08f48"
from {username='slave'} at 127.0.0.1:40566
I20260925 03:02:52.750415 23435 raft_consensus.cc:595] T 9de619025d19432c8df31a19bb6fc446 P fa551b23283b4034b256ba25acec541b [term 1 LEADER]: Received request to transfer leadership to TS b66c1d3fe3174f64b0936c512ee08f48
I20260925 03:02:52.750587 23695 consensus_queue.cc:237] T c30e9a1092fe4d99994480aab8863cf6 P 2edab657a33a4a0abfbd3262b1668483 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:02:52.760560 23328 catalog_manager.cc:5495] T c30e9a1092fe4d99994480aab8863cf6 P 2edab657a33a4a0abfbd3262b1668483 reported cstate change: term changed from 1 to 2, leader changed from b66c1d3fe3174f64b0936c512ee08f48 (127.22.193.3) to 2edab657a33a4a0abfbd3262b1668483 (127.22.193.2). New cstate: current_term: 2 leader_uuid: "2edab657a33a4a0abfbd3262b1668483" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } health_report { overall_health: UNKNOWN } } }
I20260925 03:02:52.808030 23686 consensus_queue.cc:1035] T 9de619025d19432c8df31a19bb6fc446 P fa551b23283b4034b256ba25acec541b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 03:02:52.816100 23665 raft_consensus.cc:979] T 9de619025d19432c8df31a19bb6fc446 P fa551b23283b4034b256ba25acec541b: : Instructing follower b66c1d3fe3174f64b0936c512ee08f48 to start an election
I20260925 03:02:52.816548 23665 raft_consensus.cc:1067] T 9de619025d19432c8df31a19bb6fc446 P fa551b23283b4034b256ba25acec541b [term 1 LEADER]: Signalling peer b66c1d3fe3174f64b0936c512ee08f48 to start an election
I20260925 03:02:52.819242 23584 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9de619025d19432c8df31a19bb6fc446"
dest_uuid: "b66c1d3fe3174f64b0936c512ee08f48"
from {username='slave'} at 127.0.0.1:51184
I20260925 03:02:52.819821 23584 raft_consensus.cc:482] T 9de619025d19432c8df31a19bb6fc446 P b66c1d3fe3174f64b0936c512ee08f48 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 03:02:52.820199 23584 raft_consensus.cc:3037] T 9de619025d19432c8df31a19bb6fc446 P b66c1d3fe3174f64b0936c512ee08f48 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:02:52.826004 23686 consensus_queue.cc:1035] T 9de619025d19432c8df31a19bb6fc446 P fa551b23283b4034b256ba25acec541b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 03:02:52.829803 23686 consensus_queue.cc:1035] T fb663a8b564c494f98f56845f8a05d6c P fa551b23283b4034b256ba25acec541b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260925 03:02:52.833446 23584 raft_consensus.cc:504] T 9de619025d19432c8df31a19bb6fc446 P b66c1d3fe3174f64b0936c512ee08f48 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:02:52.836285 23583 raft_consensus.cc:1226] T 9de619025d19432c8df31a19bb6fc446 P b66c1d3fe3174f64b0936c512ee08f48 [term 2 FOLLOWER]: Rejecting Update request from peer fa551b23283b4034b256ba25acec541b for earlier term 1. Current term is 2. Ops: []
I20260925 03:02:52.840381 23584 leader_election.cc:288] T 9de619025d19432c8df31a19bb6fc446 P b66c1d3fe3174f64b0936c512ee08f48 [CANDIDATE]: Term 2 election: Requested vote from peers fa551b23283b4034b256ba25acec541b (127.22.193.1:42181), 2edab657a33a4a0abfbd3262b1668483 (127.22.193.2:35431)
I20260925 03:02:52.843037 23435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9de619025d19432c8df31a19bb6fc446" candidate_uuid: "b66c1d3fe3174f64b0936c512ee08f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "fa551b23283b4034b256ba25acec541b"
I20260925 03:02:52.843688 23435 raft_consensus.cc:3032] T 9de619025d19432c8df31a19bb6fc446 P fa551b23283b4034b256ba25acec541b [term 1 LEADER]: Stepping down as leader of term 1
I20260925 03:02:52.843983 23435 raft_consensus.cc:729] T 9de619025d19432c8df31a19bb6fc446 P fa551b23283b4034b256ba25acec541b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fa551b23283b4034b256ba25acec541b, State: Running, Role: LEADER
I20260925 03:02:52.844744 23435 consensus_queue.cc:260] T 9de619025d19432c8df31a19bb6fc446 P fa551b23283b4034b256ba25acec541b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:02:52.845865 23435 raft_consensus.cc:3037] T 9de619025d19432c8df31a19bb6fc446 P fa551b23283b4034b256ba25acec541b [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:02:52.846577 23509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9de619025d19432c8df31a19bb6fc446" candidate_uuid: "b66c1d3fe3174f64b0936c512ee08f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2edab657a33a4a0abfbd3262b1668483"
I20260925 03:02:52.853569 23686 consensus_queue.cc:1035] T fb663a8b564c494f98f56845f8a05d6c P fa551b23283b4034b256ba25acec541b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260925 03:02:52.863116 23435 raft_consensus.cc:2454] T 9de619025d19432c8df31a19bb6fc446 P fa551b23283b4034b256ba25acec541b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b66c1d3fe3174f64b0936c512ee08f48 in term 2.
I20260925 03:02:52.864488 23547 leader_election.cc:302] T 9de619025d19432c8df31a19bb6fc446 P b66c1d3fe3174f64b0936c512ee08f48 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b66c1d3fe3174f64b0936c512ee08f48, fa551b23283b4034b256ba25acec541b; no voters: 2edab657a33a4a0abfbd3262b1668483
I20260925 03:02:52.865895 23704 raft_consensus.cc:2781] T 9de619025d19432c8df31a19bb6fc446 P b66c1d3fe3174f64b0936c512ee08f48 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 03:02:52.866441 23704 raft_consensus.cc:686] T 9de619025d19432c8df31a19bb6fc446 P b66c1d3fe3174f64b0936c512ee08f48 [term 2 LEADER]: Becoming Leader. State: Replica: b66c1d3fe3174f64b0936c512ee08f48, State: Running, Role: LEADER
I20260925 03:02:52.867318 23704 consensus_queue.cc:237] T 9de619025d19432c8df31a19bb6fc446 P b66c1d3fe3174f64b0936c512ee08f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:02:52.888792 23706 raft_consensus.cc:979] T fb663a8b564c494f98f56845f8a05d6c P fa551b23283b4034b256ba25acec541b: : Instructing follower 2edab657a33a4a0abfbd3262b1668483 to start an election
I20260925 03:02:52.889274 23665 raft_consensus.cc:1067] T fb663a8b564c494f98f56845f8a05d6c P fa551b23283b4034b256ba25acec541b [term 1 LEADER]: Signalling peer 2edab657a33a4a0abfbd3262b1668483 to start an election
I20260925 03:02:52.886809 23331 catalog_manager.cc:5495] T 9de619025d19432c8df31a19bb6fc446 P b66c1d3fe3174f64b0936c512ee08f48 reported cstate change: term changed from 1 to 2, leader changed from fa551b23283b4034b256ba25acec541b (127.22.193.1) to b66c1d3fe3174f64b0936c512ee08f48 (127.22.193.3). New cstate: current_term: 2 leader_uuid: "b66c1d3fe3174f64b0936c512ee08f48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } health_report { overall_health: HEALTHY } } }
I20260925 03:02:52.905005 23508 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fb663a8b564c494f98f56845f8a05d6c"
dest_uuid: "2edab657a33a4a0abfbd3262b1668483"
from {username='slave'} at 127.0.0.1:38974
I20260925 03:02:52.905607 23508 raft_consensus.cc:482] T fb663a8b564c494f98f56845f8a05d6c P 2edab657a33a4a0abfbd3262b1668483 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 03:02:52.906028 23508 raft_consensus.cc:3037] T fb663a8b564c494f98f56845f8a05d6c P 2edab657a33a4a0abfbd3262b1668483 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:02:52.912966 23508 raft_consensus.cc:504] T fb663a8b564c494f98f56845f8a05d6c P 2edab657a33a4a0abfbd3262b1668483 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:02:52.914987 23508 leader_election.cc:288] T fb663a8b564c494f98f56845f8a05d6c P 2edab657a33a4a0abfbd3262b1668483 [CANDIDATE]: Term 2 election: Requested vote from peers fa551b23283b4034b256ba25acec541b (127.22.193.1:42181), b66c1d3fe3174f64b0936c512ee08f48 (127.22.193.3:42621)
I20260925 03:02:52.916913 23582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb663a8b564c494f98f56845f8a05d6c" candidate_uuid: "2edab657a33a4a0abfbd3262b1668483" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b66c1d3fe3174f64b0936c512ee08f48"
I20260925 03:02:52.916839 23435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb663a8b564c494f98f56845f8a05d6c" candidate_uuid: "2edab657a33a4a0abfbd3262b1668483" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "fa551b23283b4034b256ba25acec541b"
I20260925 03:02:52.917534 23582 raft_consensus.cc:3037] T fb663a8b564c494f98f56845f8a05d6c P b66c1d3fe3174f64b0936c512ee08f48 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:02:52.917647 23435 raft_consensus.cc:3032] T fb663a8b564c494f98f56845f8a05d6c P fa551b23283b4034b256ba25acec541b [term 1 LEADER]: Stepping down as leader of term 1
I20260925 03:02:52.918073 23435 raft_consensus.cc:729] T fb663a8b564c494f98f56845f8a05d6c P fa551b23283b4034b256ba25acec541b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fa551b23283b4034b256ba25acec541b, State: Running, Role: LEADER
I20260925 03:02:52.918861 23435 consensus_queue.cc:260] T fb663a8b564c494f98f56845f8a05d6c P fa551b23283b4034b256ba25acec541b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:02:52.920126 23435 raft_consensus.cc:3037] T fb663a8b564c494f98f56845f8a05d6c P fa551b23283b4034b256ba25acec541b [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:02:52.923369 23582 raft_consensus.cc:2454] T fb663a8b564c494f98f56845f8a05d6c P b66c1d3fe3174f64b0936c512ee08f48 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2edab657a33a4a0abfbd3262b1668483 in term 2.
I20260925 03:02:52.924606 23473 leader_election.cc:302] T fb663a8b564c494f98f56845f8a05d6c P 2edab657a33a4a0abfbd3262b1668483 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2edab657a33a4a0abfbd3262b1668483, b66c1d3fe3174f64b0936c512ee08f48; no voters:
I20260925 03:02:52.925405 23695 raft_consensus.cc:2781] T fb663a8b564c494f98f56845f8a05d6c P 2edab657a33a4a0abfbd3262b1668483 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 03:02:52.926049 23695 raft_consensus.cc:686] T fb663a8b564c494f98f56845f8a05d6c P 2edab657a33a4a0abfbd3262b1668483 [term 2 LEADER]: Becoming Leader. State: Replica: 2edab657a33a4a0abfbd3262b1668483, State: Running, Role: LEADER
I20260925 03:02:52.926628 23435 raft_consensus.cc:2454] T fb663a8b564c494f98f56845f8a05d6c P fa551b23283b4034b256ba25acec541b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2edab657a33a4a0abfbd3262b1668483 in term 2.
I20260925 03:02:52.926828 23695 consensus_queue.cc:237] T fb663a8b564c494f98f56845f8a05d6c P 2edab657a33a4a0abfbd3262b1668483 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:02:52.932560 23704 consensus_queue.cc:1035] T 138dfa413a144a0e8911a2b402857cb4 P b66c1d3fe3174f64b0936c512ee08f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 03:02:52.936946 23331 catalog_manager.cc:5495] T fb663a8b564c494f98f56845f8a05d6c P 2edab657a33a4a0abfbd3262b1668483 reported cstate change: term changed from 1 to 2, leader changed from fa551b23283b4034b256ba25acec541b (127.22.193.1) to 2edab657a33a4a0abfbd3262b1668483 (127.22.193.2). New cstate: current_term: 2 leader_uuid: "2edab657a33a4a0abfbd3262b1668483" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } health_report { overall_health: UNKNOWN } } }
I20260925 03:02:52.943506 23634 raft_consensus.cc:979] T 138dfa413a144a0e8911a2b402857cb4 P b66c1d3fe3174f64b0936c512ee08f48: : Instructing follower 2edab657a33a4a0abfbd3262b1668483 to start an election
I20260925 03:02:52.944166 23704 raft_consensus.cc:1067] T 138dfa413a144a0e8911a2b402857cb4 P b66c1d3fe3174f64b0936c512ee08f48 [term 1 LEADER]: Signalling peer 2edab657a33a4a0abfbd3262b1668483 to start an election
I20260925 03:02:52.946964 23509 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "138dfa413a144a0e8911a2b402857cb4"
dest_uuid: "2edab657a33a4a0abfbd3262b1668483"
from {username='slave'} at 127.0.0.1:38970
I20260925 03:02:52.947620 23509 raft_consensus.cc:482] T 138dfa413a144a0e8911a2b402857cb4 P 2edab657a33a4a0abfbd3262b1668483 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 03:02:52.948001 23509 raft_consensus.cc:3037] T 138dfa413a144a0e8911a2b402857cb4 P 2edab657a33a4a0abfbd3262b1668483 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:02:52.954948 23682 consensus_queue.cc:1035] T 138dfa413a144a0e8911a2b402857cb4 P b66c1d3fe3174f64b0936c512ee08f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 03:02:52.957324 23509 raft_consensus.cc:504] T 138dfa413a144a0e8911a2b402857cb4 P 2edab657a33a4a0abfbd3262b1668483 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:02:52.958920 23508 raft_consensus.cc:1226] T 138dfa413a144a0e8911a2b402857cb4 P 2edab657a33a4a0abfbd3262b1668483 [term 2 FOLLOWER]: Rejecting Update request from peer b66c1d3fe3174f64b0936c512ee08f48 for earlier term 1. Current term is 2. Ops: []
I20260925 03:02:52.963172 23637 raft_consensus.cc:979] T ef74cafe53e34e1dace8b02d73693521 P 2edab657a33a4a0abfbd3262b1668483: : Instructing follower fa551b23283b4034b256ba25acec541b to start an election
I20260925 03:02:52.963927 23637 raft_consensus.cc:1067] T ef74cafe53e34e1dace8b02d73693521 P 2edab657a33a4a0abfbd3262b1668483 [term 1 LEADER]: Signalling peer fa551b23283b4034b256ba25acec541b to start an election
I20260925 03:02:52.966332 23435 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ef74cafe53e34e1dace8b02d73693521"
dest_uuid: "fa551b23283b4034b256ba25acec541b"
from {username='slave'} at 127.0.0.1:40538
I20260925 03:02:52.967073 23435 raft_consensus.cc:482] T ef74cafe53e34e1dace8b02d73693521 P fa551b23283b4034b256ba25acec541b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 03:02:52.967576 23435 raft_consensus.cc:3037] T ef74cafe53e34e1dace8b02d73693521 P fa551b23283b4034b256ba25acec541b [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:02:52.969429 23433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "138dfa413a144a0e8911a2b402857cb4" candidate_uuid: "2edab657a33a4a0abfbd3262b1668483" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "fa551b23283b4034b256ba25acec541b"
I20260925 03:02:52.969110 23634 consensus_queue.cc:1046] T 138dfa413a144a0e8911a2b402857cb4 P b66c1d3fe3174f64b0936c512ee08f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.002s
I20260925 03:02:52.971300 23634 raft_consensus.cc:3032] T 138dfa413a144a0e8911a2b402857cb4 P b66c1d3fe3174f64b0936c512ee08f48 [term 1 LEADER]: Stepping down as leader of term 1
I20260925 03:02:52.971614 23634 raft_consensus.cc:729] T 138dfa413a144a0e8911a2b402857cb4 P b66c1d3fe3174f64b0936c512ee08f48 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b66c1d3fe3174f64b0936c512ee08f48, State: Running, Role: LEADER
I20260925 03:02:52.972079 23509 leader_election.cc:288] T 138dfa413a144a0e8911a2b402857cb4 P 2edab657a33a4a0abfbd3262b1668483 [CANDIDATE]: Term 2 election: Requested vote from peers fa551b23283b4034b256ba25acec541b (127.22.193.1:42181), b66c1d3fe3174f64b0936c512ee08f48 (127.22.193.3:42621)
I20260925 03:02:52.972558 23582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "138dfa413a144a0e8911a2b402857cb4" candidate_uuid: "2edab657a33a4a0abfbd3262b1668483" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b66c1d3fe3174f64b0936c512ee08f48"
I20260925 03:02:52.976610 23634 consensus_queue.cc:260] T 138dfa413a144a0e8911a2b402857cb4 P b66c1d3fe3174f64b0936c512ee08f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:02:52.982254 23634 raft_consensus.cc:3037] T 138dfa413a144a0e8911a2b402857cb4 P b66c1d3fe3174f64b0936c512ee08f48 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:02:52.982580 23435 raft_consensus.cc:504] T ef74cafe53e34e1dace8b02d73693521 P fa551b23283b4034b256ba25acec541b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } } peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } }
I20260925 03:02:52.984966 23435 leader_election.cc:288] T ef74cafe53e34e1dace8b02d73693521 P fa551b23283b4034b256ba25acec541b [CANDIDATE]: Term 2 election: Requested vote from peers b66c1d3fe3174f64b0936c512ee08f48 (127.22.193.3:42621), 2edab657a33a4a0abfbd3262b1668483 (127.22.193.2:35431)
I20260925 03:02:52.986636 23509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef74cafe53e34e1dace8b02d73693521" candidate_uuid: "fa551b23283b4034b256ba25acec541b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2edab657a33a4a0abfbd3262b1668483"
I20260925 03:02:52.987277 23509 raft_consensus.cc:3032] T ef74cafe53e34e1dace8b02d73693521 P 2edab657a33a4a0abfbd3262b1668483 [term 1 LEADER]: Stepping down as leader of term 1
I20260925 03:02:52.987206 23584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef74cafe53e34e1dace8b02d73693521" candidate_uuid: "fa551b23283b4034b256ba25acec541b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b66c1d3fe3174f64b0936c512ee08f48"
I20260925 03:02:52.987735 23509 raft_consensus.cc:729] T ef74cafe53e34e1dace8b02d73693521 P 2edab657a33a4a0abfbd3262b1668483 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2edab657a33a4a0abfbd3262b1668483, State: Running, Role: LEADER
I20260925 03:02:52.988220 23584 raft_consensus.cc:3037] T ef74cafe53e34e1dace8b02d73693521 P b66c1d3fe3174f64b0936c512ee08f48 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:02:52.988518 23509 consensus_queue.cc:260] T ef74cafe53e34e1dace8b02d73693521 P 2edab657a33a4a0abfbd3262b1668483 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } } peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } }
I20260925 03:02:52.989934 23509 raft_consensus.cc:3037] T ef74cafe53e34e1dace8b02d73693521 P 2edab657a33a4a0abfbd3262b1668483 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:02:52.994665 23584 raft_consensus.cc:2454] T ef74cafe53e34e1dace8b02d73693521 P b66c1d3fe3174f64b0936c512ee08f48 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa551b23283b4034b256ba25acec541b in term 2.
I20260925 03:02:52.996017 23398 leader_election.cc:302] T ef74cafe53e34e1dace8b02d73693521 P fa551b23283b4034b256ba25acec541b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b66c1d3fe3174f64b0936c512ee08f48, fa551b23283b4034b256ba25acec541b; no voters:
I20260925 03:02:52.996454 23509 raft_consensus.cc:2454] T ef74cafe53e34e1dace8b02d73693521 P 2edab657a33a4a0abfbd3262b1668483 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa551b23283b4034b256ba25acec541b in term 2.
I20260925 03:02:52.996953 23686 raft_consensus.cc:2781] T ef74cafe53e34e1dace8b02d73693521 P fa551b23283b4034b256ba25acec541b [term 2 FOLLOWER]: Leader election won for term 2
I20260925 03:02:52.997426 23686 raft_consensus.cc:686] T ef74cafe53e34e1dace8b02d73693521 P fa551b23283b4034b256ba25acec541b [term 2 LEADER]: Becoming Leader. State: Replica: fa551b23283b4034b256ba25acec541b, State: Running, Role: LEADER
I20260925 03:02:52.998216 23686 consensus_queue.cc:237] T ef74cafe53e34e1dace8b02d73693521 P fa551b23283b4034b256ba25acec541b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } } peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } }
I20260925 03:02:53.002403 23582 raft_consensus.cc:2454] T 138dfa413a144a0e8911a2b402857cb4 P b66c1d3fe3174f64b0936c512ee08f48 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2edab657a33a4a0abfbd3262b1668483 in term 2.
I20260925 03:02:53.004667 23473 leader_election.cc:302] T 138dfa413a144a0e8911a2b402857cb4 P 2edab657a33a4a0abfbd3262b1668483 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2edab657a33a4a0abfbd3262b1668483, b66c1d3fe3174f64b0936c512ee08f48; no voters: fa551b23283b4034b256ba25acec541b
I20260925 03:02:53.005867 23695 raft_consensus.cc:2781] T 138dfa413a144a0e8911a2b402857cb4 P 2edab657a33a4a0abfbd3262b1668483 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 03:02:53.006325 23695 raft_consensus.cc:686] T 138dfa413a144a0e8911a2b402857cb4 P 2edab657a33a4a0abfbd3262b1668483 [term 2 LEADER]: Becoming Leader. State: Replica: 2edab657a33a4a0abfbd3262b1668483, State: Running, Role: LEADER
I20260925 03:02:53.007076 23695 consensus_queue.cc:237] T 138dfa413a144a0e8911a2b402857cb4 P 2edab657a33a4a0abfbd3262b1668483 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:02:53.012517 23328 catalog_manager.cc:5495] T ef74cafe53e34e1dace8b02d73693521 P fa551b23283b4034b256ba25acec541b reported cstate change: term changed from 1 to 2, leader changed from 2edab657a33a4a0abfbd3262b1668483 (127.22.193.2) to fa551b23283b4034b256ba25acec541b (127.22.193.1). New cstate: current_term: 2 leader_uuid: "fa551b23283b4034b256ba25acec541b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } health_report { overall_health: UNKNOWN } } }
I20260925 03:02:53.017172 23331 catalog_manager.cc:5495] T 138dfa413a144a0e8911a2b402857cb4 P 2edab657a33a4a0abfbd3262b1668483 reported cstate change: term changed from 1 to 2, leader changed from b66c1d3fe3174f64b0936c512ee08f48 (127.22.193.3) to 2edab657a33a4a0abfbd3262b1668483 (127.22.193.2). New cstate: current_term: 2 leader_uuid: "2edab657a33a4a0abfbd3262b1668483" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } health_report { overall_health: UNKNOWN } } }
I20260925 03:02:53.029552 23660 raft_consensus.cc:979] T d01220d66a8c4d34ac86fc3559fa5165 P 2edab657a33a4a0abfbd3262b1668483: : Instructing follower fa551b23283b4034b256ba25acec541b to start an election
I20260925 03:02:53.030016 23695 raft_consensus.cc:1067] T d01220d66a8c4d34ac86fc3559fa5165 P 2edab657a33a4a0abfbd3262b1668483 [term 1 LEADER]: Signalling peer fa551b23283b4034b256ba25acec541b to start an election
I20260925 03:02:53.032011 23435 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d01220d66a8c4d34ac86fc3559fa5165"
dest_uuid: "fa551b23283b4034b256ba25acec541b"
from {username='slave'} at 127.0.0.1:40538
I20260925 03:02:53.032532 23435 raft_consensus.cc:482] T d01220d66a8c4d34ac86fc3559fa5165 P fa551b23283b4034b256ba25acec541b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 03:02:53.032867 23435 raft_consensus.cc:3037] T d01220d66a8c4d34ac86fc3559fa5165 P fa551b23283b4034b256ba25acec541b [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:02:53.040217 23435 raft_consensus.cc:504] T d01220d66a8c4d34ac86fc3559fa5165 P fa551b23283b4034b256ba25acec541b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:02:53.041805 23435 leader_election.cc:288] T d01220d66a8c4d34ac86fc3559fa5165 P fa551b23283b4034b256ba25acec541b [CANDIDATE]: Term 2 election: Requested vote from peers 2edab657a33a4a0abfbd3262b1668483 (127.22.193.2:35431), b66c1d3fe3174f64b0936c512ee08f48 (127.22.193.3:42621)
I20260925 03:02:53.043007 23509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d01220d66a8c4d34ac86fc3559fa5165" candidate_uuid: "fa551b23283b4034b256ba25acec541b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2edab657a33a4a0abfbd3262b1668483"
I20260925 03:02:53.042997 23582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d01220d66a8c4d34ac86fc3559fa5165" candidate_uuid: "fa551b23283b4034b256ba25acec541b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b66c1d3fe3174f64b0936c512ee08f48"
I20260925 03:02:53.043789 23509 raft_consensus.cc:3032] T d01220d66a8c4d34ac86fc3559fa5165 P 2edab657a33a4a0abfbd3262b1668483 [term 1 LEADER]: Stepping down as leader of term 1
I20260925 03:02:53.043864 23582 raft_consensus.cc:3037] T d01220d66a8c4d34ac86fc3559fa5165 P b66c1d3fe3174f64b0936c512ee08f48 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:02:53.044229 23509 raft_consensus.cc:729] T d01220d66a8c4d34ac86fc3559fa5165 P 2edab657a33a4a0abfbd3262b1668483 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2edab657a33a4a0abfbd3262b1668483, State: Running, Role: LEADER
I20260925 03:02:53.045019 23509 consensus_queue.cc:260] T d01220d66a8c4d34ac86fc3559fa5165 P 2edab657a33a4a0abfbd3262b1668483 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:02:53.046353 23509 raft_consensus.cc:3037] T d01220d66a8c4d34ac86fc3559fa5165 P 2edab657a33a4a0abfbd3262b1668483 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:02:53.049151 23634 consensus_queue.cc:1035] T 832e06098a3e499f82d53853e26235cf P b66c1d3fe3174f64b0936c512ee08f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 03:02:53.050576 23582 raft_consensus.cc:2454] T d01220d66a8c4d34ac86fc3559fa5165 P b66c1d3fe3174f64b0936c512ee08f48 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa551b23283b4034b256ba25acec541b in term 2.
I20260925 03:02:53.051695 23398 leader_election.cc:302] T d01220d66a8c4d34ac86fc3559fa5165 P fa551b23283b4034b256ba25acec541b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b66c1d3fe3174f64b0936c512ee08f48, fa551b23283b4034b256ba25acec541b; no voters:
I20260925 03:02:53.052610 23686 raft_consensus.cc:2781] T d01220d66a8c4d34ac86fc3559fa5165 P fa551b23283b4034b256ba25acec541b [term 2 FOLLOWER]: Leader election won for term 2
I20260925 03:02:53.053154 23686 raft_consensus.cc:686] T d01220d66a8c4d34ac86fc3559fa5165 P fa551b23283b4034b256ba25acec541b [term 2 LEADER]: Becoming Leader. State: Replica: fa551b23283b4034b256ba25acec541b, State: Running, Role: LEADER
I20260925 03:02:53.054013 23686 consensus_queue.cc:237] T d01220d66a8c4d34ac86fc3559fa5165 P fa551b23283b4034b256ba25acec541b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:02:53.059306 23509 raft_consensus.cc:2454] T d01220d66a8c4d34ac86fc3559fa5165 P 2edab657a33a4a0abfbd3262b1668483 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa551b23283b4034b256ba25acec541b in term 2.
I20260925 03:02:53.065977 23682 consensus_queue.cc:1035] T 832e06098a3e499f82d53853e26235cf P b66c1d3fe3174f64b0936c512ee08f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260925 03:02:53.072847 23332 catalog_manager.cc:5495] T d01220d66a8c4d34ac86fc3559fa5165 P fa551b23283b4034b256ba25acec541b reported cstate change: term changed from 1 to 2, leader changed from 2edab657a33a4a0abfbd3262b1668483 (127.22.193.2) to fa551b23283b4034b256ba25acec541b (127.22.193.1). New cstate: current_term: 2 leader_uuid: "fa551b23283b4034b256ba25acec541b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } health_report { overall_health: UNKNOWN } } }
I20260925 03:02:53.078737 23634 raft_consensus.cc:979] T 832e06098a3e499f82d53853e26235cf P b66c1d3fe3174f64b0936c512ee08f48: : Instructing follower fa551b23283b4034b256ba25acec541b to start an election
I20260925 03:02:53.079226 23634 raft_consensus.cc:1067] T 832e06098a3e499f82d53853e26235cf P b66c1d3fe3174f64b0936c512ee08f48 [term 1 LEADER]: Signalling peer fa551b23283b4034b256ba25acec541b to start an election
I20260925 03:02:53.080920 23435 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "832e06098a3e499f82d53853e26235cf"
dest_uuid: "fa551b23283b4034b256ba25acec541b"
from {username='slave'} at 127.0.0.1:40552
I20260925 03:02:53.082535 23435 raft_consensus.cc:482] T 832e06098a3e499f82d53853e26235cf P fa551b23283b4034b256ba25acec541b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 03:02:53.083369 23435 raft_consensus.cc:3037] T 832e06098a3e499f82d53853e26235cf P fa551b23283b4034b256ba25acec541b [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:02:53.091729 23435 raft_consensus.cc:504] T 832e06098a3e499f82d53853e26235cf P fa551b23283b4034b256ba25acec541b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:02:53.094089 23435 leader_election.cc:288] T 832e06098a3e499f82d53853e26235cf P fa551b23283b4034b256ba25acec541b [CANDIDATE]: Term 2 election: Requested vote from peers 2edab657a33a4a0abfbd3262b1668483 (127.22.193.2:35431), b66c1d3fe3174f64b0936c512ee08f48 (127.22.193.3:42621)
I20260925 03:02:53.094676 23508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "832e06098a3e499f82d53853e26235cf" candidate_uuid: "fa551b23283b4034b256ba25acec541b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2edab657a33a4a0abfbd3262b1668483"
I20260925 03:02:53.095295 23508 raft_consensus.cc:3037] T 832e06098a3e499f82d53853e26235cf P 2edab657a33a4a0abfbd3262b1668483 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:02:53.095422 23582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "832e06098a3e499f82d53853e26235cf" candidate_uuid: "fa551b23283b4034b256ba25acec541b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b66c1d3fe3174f64b0936c512ee08f48"
I20260925 03:02:53.096251 23582 raft_consensus.cc:3032] T 832e06098a3e499f82d53853e26235cf P b66c1d3fe3174f64b0936c512ee08f48 [term 1 LEADER]: Stepping down as leader of term 1
I20260925 03:02:53.096881 23582 raft_consensus.cc:729] T 832e06098a3e499f82d53853e26235cf P b66c1d3fe3174f64b0936c512ee08f48 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b66c1d3fe3174f64b0936c512ee08f48, State: Running, Role: LEADER
I20260925 03:02:53.097999 23582 consensus_queue.cc:260] T 832e06098a3e499f82d53853e26235cf P b66c1d3fe3174f64b0936c512ee08f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:02:53.099167 23582 raft_consensus.cc:3037] T 832e06098a3e499f82d53853e26235cf P b66c1d3fe3174f64b0936c512ee08f48 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:02:53.104079 23508 raft_consensus.cc:2454] T 832e06098a3e499f82d53853e26235cf P 2edab657a33a4a0abfbd3262b1668483 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa551b23283b4034b256ba25acec541b in term 2.
I20260925 03:02:53.105505 23399 leader_election.cc:302] T 832e06098a3e499f82d53853e26235cf P fa551b23283b4034b256ba25acec541b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2edab657a33a4a0abfbd3262b1668483, fa551b23283b4034b256ba25acec541b; no voters:
I20260925 03:02:53.106341 23686 raft_consensus.cc:2781] T 832e06098a3e499f82d53853e26235cf P fa551b23283b4034b256ba25acec541b [term 2 FOLLOWER]: Leader election won for term 2
I20260925 03:02:53.106503 23582 raft_consensus.cc:2454] T 832e06098a3e499f82d53853e26235cf P b66c1d3fe3174f64b0936c512ee08f48 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa551b23283b4034b256ba25acec541b in term 2.
I20260925 03:02:53.106863 23686 raft_consensus.cc:686] T 832e06098a3e499f82d53853e26235cf P fa551b23283b4034b256ba25acec541b [term 2 LEADER]: Becoming Leader. State: Replica: fa551b23283b4034b256ba25acec541b, State: Running, Role: LEADER
I20260925 03:02:53.107781 23686 consensus_queue.cc:237] T 832e06098a3e499f82d53853e26235cf P fa551b23283b4034b256ba25acec541b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:02:53.115283 23331 catalog_manager.cc:5495] T 832e06098a3e499f82d53853e26235cf P fa551b23283b4034b256ba25acec541b reported cstate change: term changed from 1 to 2, leader changed from b66c1d3fe3174f64b0936c512ee08f48 (127.22.193.3) to fa551b23283b4034b256ba25acec541b (127.22.193.1). New cstate: current_term: 2 leader_uuid: "fa551b23283b4034b256ba25acec541b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } health_report { overall_health: UNKNOWN } } }
I20260925 03:02:53.171813 23660 raft_consensus.cc:979] T 2baf62cd63564f4ba08b201e40e19d55 P 2edab657a33a4a0abfbd3262b1668483: : Instructing follower fa551b23283b4034b256ba25acec541b to start an election
I20260925 03:02:53.172397 23695 raft_consensus.cc:1067] T 2baf62cd63564f4ba08b201e40e19d55 P 2edab657a33a4a0abfbd3262b1668483 [term 1 LEADER]: Signalling peer fa551b23283b4034b256ba25acec541b to start an election
I20260925 03:02:53.174036 23435 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2baf62cd63564f4ba08b201e40e19d55"
dest_uuid: "fa551b23283b4034b256ba25acec541b"
from {username='slave'} at 127.0.0.1:40538
I20260925 03:02:53.174837 23435 raft_consensus.cc:482] T 2baf62cd63564f4ba08b201e40e19d55 P fa551b23283b4034b256ba25acec541b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 03:02:53.175133 23435 raft_consensus.cc:3037] T 2baf62cd63564f4ba08b201e40e19d55 P fa551b23283b4034b256ba25acec541b [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:02:53.180851 23435 raft_consensus.cc:504] T 2baf62cd63564f4ba08b201e40e19d55 P fa551b23283b4034b256ba25acec541b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:02:53.182530 23435 leader_election.cc:288] T 2baf62cd63564f4ba08b201e40e19d55 P fa551b23283b4034b256ba25acec541b [CANDIDATE]: Term 2 election: Requested vote from peers 2edab657a33a4a0abfbd3262b1668483 (127.22.193.2:35431), b66c1d3fe3174f64b0936c512ee08f48 (127.22.193.3:42621)
I20260925 03:02:53.183776 23508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2baf62cd63564f4ba08b201e40e19d55" candidate_uuid: "fa551b23283b4034b256ba25acec541b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2edab657a33a4a0abfbd3262b1668483"
I20260925 03:02:53.184547 23508 raft_consensus.cc:3032] T 2baf62cd63564f4ba08b201e40e19d55 P 2edab657a33a4a0abfbd3262b1668483 [term 1 LEADER]: Stepping down as leader of term 1
I20260925 03:02:53.184859 23508 raft_consensus.cc:729] T 2baf62cd63564f4ba08b201e40e19d55 P 2edab657a33a4a0abfbd3262b1668483 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2edab657a33a4a0abfbd3262b1668483, State: Running, Role: LEADER
I20260925 03:02:53.185462 23508 consensus_queue.cc:260] T 2baf62cd63564f4ba08b201e40e19d55 P 2edab657a33a4a0abfbd3262b1668483 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:02:53.186676 23508 raft_consensus.cc:3037] T 2baf62cd63564f4ba08b201e40e19d55 P 2edab657a33a4a0abfbd3262b1668483 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:02:53.183746 23582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2baf62cd63564f4ba08b201e40e19d55" candidate_uuid: "fa551b23283b4034b256ba25acec541b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b66c1d3fe3174f64b0936c512ee08f48"
I20260925 03:02:53.187397 23582 raft_consensus.cc:3037] T 2baf62cd63564f4ba08b201e40e19d55 P b66c1d3fe3174f64b0936c512ee08f48 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:02:53.194131 23582 raft_consensus.cc:2454] T 2baf62cd63564f4ba08b201e40e19d55 P b66c1d3fe3174f64b0936c512ee08f48 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa551b23283b4034b256ba25acec541b in term 2.
I20260925 03:02:53.194131 23508 raft_consensus.cc:2454] T 2baf62cd63564f4ba08b201e40e19d55 P 2edab657a33a4a0abfbd3262b1668483 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa551b23283b4034b256ba25acec541b in term 2.
I20260925 03:02:53.195501 23398 leader_election.cc:302] T 2baf62cd63564f4ba08b201e40e19d55 P fa551b23283b4034b256ba25acec541b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b66c1d3fe3174f64b0936c512ee08f48, fa551b23283b4034b256ba25acec541b; no voters:
I20260925 03:02:53.196177 23686 raft_consensus.cc:2781] T 2baf62cd63564f4ba08b201e40e19d55 P fa551b23283b4034b256ba25acec541b [term 2 FOLLOWER]: Leader election won for term 2
I20260925 03:02:53.196630 23686 raft_consensus.cc:686] T 2baf62cd63564f4ba08b201e40e19d55 P fa551b23283b4034b256ba25acec541b [term 2 LEADER]: Becoming Leader. State: Replica: fa551b23283b4034b256ba25acec541b, State: Running, Role: LEADER
I20260925 03:02:53.197372 23686 consensus_queue.cc:237] T 2baf62cd63564f4ba08b201e40e19d55 P fa551b23283b4034b256ba25acec541b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:02:53.204171 23331 catalog_manager.cc:5495] T 2baf62cd63564f4ba08b201e40e19d55 P fa551b23283b4034b256ba25acec541b reported cstate change: term changed from 1 to 2, leader changed from 2edab657a33a4a0abfbd3262b1668483 (127.22.193.2) to fa551b23283b4034b256ba25acec541b (127.22.193.1). New cstate: current_term: 2 leader_uuid: "fa551b23283b4034b256ba25acec541b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } health_report { overall_health: UNKNOWN } } }
I20260925 03:02:53.222805 23582 raft_consensus.cc:1261] T c30e9a1092fe4d99994480aab8863cf6 P b66c1d3fe3174f64b0936c512ee08f48 [term 2 FOLLOWER]: Refusing update from remote peer 2edab657a33a4a0abfbd3262b1668483: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260925 03:02:53.224447 23695 consensus_queue.cc:1035] T c30e9a1092fe4d99994480aab8863cf6 P 2edab657a33a4a0abfbd3262b1668483 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260925 03:02:53.236299 23435 raft_consensus.cc:1261] T c30e9a1092fe4d99994480aab8863cf6 P fa551b23283b4034b256ba25acec541b [term 2 FOLLOWER]: Refusing update from remote peer 2edab657a33a4a0abfbd3262b1668483: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260925 03:02:53.238656 23637 consensus_queue.cc:1035] T c30e9a1092fe4d99994480aab8863cf6 P 2edab657a33a4a0abfbd3262b1668483 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 03:02:53.407024 23582 raft_consensus.cc:1261] T 138dfa413a144a0e8911a2b402857cb4 P b66c1d3fe3174f64b0936c512ee08f48 [term 2 FOLLOWER]: Refusing update from remote peer 2edab657a33a4a0abfbd3262b1668483: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260925 03:02:53.408329 23637 consensus_queue.cc:1035] T 138dfa413a144a0e8911a2b402857cb4 P 2edab657a33a4a0abfbd3262b1668483 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260925 03:02:53.411801 23508 raft_consensus.cc:3037] T 9de619025d19432c8df31a19bb6fc446 P 2edab657a33a4a0abfbd3262b1668483 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:02:53.418251 23508 raft_consensus.cc:1261] T 9de619025d19432c8df31a19bb6fc446 P 2edab657a33a4a0abfbd3262b1668483 [term 2 FOLLOWER]: Refusing update from remote peer b66c1d3fe3174f64b0936c512ee08f48: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260925 03:02:53.419656 23435 raft_consensus.cc:3037] T 138dfa413a144a0e8911a2b402857cb4 P fa551b23283b4034b256ba25acec541b [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:02:53.420413 23704 consensus_queue.cc:1035] T 9de619025d19432c8df31a19bb6fc446 P b66c1d3fe3174f64b0936c512ee08f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260925 03:02:53.426378 23435 raft_consensus.cc:1261] T 138dfa413a144a0e8911a2b402857cb4 P fa551b23283b4034b256ba25acec541b [term 2 FOLLOWER]: Refusing update from remote peer 2edab657a33a4a0abfbd3262b1668483: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260925 03:02:53.428320 23637 consensus_queue.cc:1035] T 138dfa413a144a0e8911a2b402857cb4 P 2edab657a33a4a0abfbd3262b1668483 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260925 03:02:53.429229 23435 raft_consensus.cc:1261] T 9de619025d19432c8df31a19bb6fc446 P fa551b23283b4034b256ba25acec541b [term 2 FOLLOWER]: Refusing update from remote peer b66c1d3fe3174f64b0936c512ee08f48: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260925 03:02:53.430817 23682 consensus_queue.cc:1035] T 9de619025d19432c8df31a19bb6fc446 P b66c1d3fe3174f64b0936c512ee08f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260925 03:02:53.440210 23508 raft_consensus.cc:1261] T d01220d66a8c4d34ac86fc3559fa5165 P 2edab657a33a4a0abfbd3262b1668483 [term 2 FOLLOWER]: Refusing update from remote peer fa551b23283b4034b256ba25acec541b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260925 03:02:53.441560 23686 consensus_queue.cc:1035] T d01220d66a8c4d34ac86fc3559fa5165 P fa551b23283b4034b256ba25acec541b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260925 03:02:53.450374 23582 raft_consensus.cc:1261] T d01220d66a8c4d34ac86fc3559fa5165 P b66c1d3fe3174f64b0936c512ee08f48 [term 2 FOLLOWER]: Refusing update from remote peer fa551b23283b4034b256ba25acec541b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260925 03:02:53.452039 23686 consensus_queue.cc:1035] T d01220d66a8c4d34ac86fc3559fa5165 P fa551b23283b4034b256ba25acec541b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260925 03:02:53.455588 23584 raft_consensus.cc:1261] T fb663a8b564c494f98f56845f8a05d6c P b66c1d3fe3174f64b0936c512ee08f48 [term 2 FOLLOWER]: Refusing update from remote peer 2edab657a33a4a0abfbd3262b1668483: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260925 03:02:53.458171 23637 consensus_queue.cc:1035] T fb663a8b564c494f98f56845f8a05d6c P 2edab657a33a4a0abfbd3262b1668483 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260925 03:02:53.468135 23434 raft_consensus.cc:1261] T fb663a8b564c494f98f56845f8a05d6c P fa551b23283b4034b256ba25acec541b [term 2 FOLLOWER]: Refusing update from remote peer 2edab657a33a4a0abfbd3262b1668483: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260925 03:02:53.470641 23637 consensus_queue.cc:1035] T fb663a8b564c494f98f56845f8a05d6c P 2edab657a33a4a0abfbd3262b1668483 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260925 03:02:53.496910 23508 raft_consensus.cc:1261] T 832e06098a3e499f82d53853e26235cf P 2edab657a33a4a0abfbd3262b1668483 [term 2 FOLLOWER]: Refusing update from remote peer fa551b23283b4034b256ba25acec541b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260925 03:02:53.498118 23686 consensus_queue.cc:1035] T 832e06098a3e499f82d53853e26235cf P fa551b23283b4034b256ba25acec541b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260925 03:02:53.506302 23582 raft_consensus.cc:1261] T 832e06098a3e499f82d53853e26235cf P b66c1d3fe3174f64b0936c512ee08f48 [term 2 FOLLOWER]: Refusing update from remote peer fa551b23283b4034b256ba25acec541b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260925 03:02:53.507639 23665 consensus_queue.cc:1035] T 832e06098a3e499f82d53853e26235cf P fa551b23283b4034b256ba25acec541b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260925 03:02:53.518338 23508 raft_consensus.cc:1261] T ef74cafe53e34e1dace8b02d73693521 P 2edab657a33a4a0abfbd3262b1668483 [term 2 FOLLOWER]: Refusing update from remote peer fa551b23283b4034b256ba25acec541b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260925 03:02:53.519486 23665 consensus_queue.cc:1035] T ef74cafe53e34e1dace8b02d73693521 P fa551b23283b4034b256ba25acec541b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260925 03:02:53.527864 23582 raft_consensus.cc:1261] T ef74cafe53e34e1dace8b02d73693521 P b66c1d3fe3174f64b0936c512ee08f48 [term 2 FOLLOWER]: Refusing update from remote peer fa551b23283b4034b256ba25acec541b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260925 03:02:53.529119 23665 consensus_queue.cc:1035] T ef74cafe53e34e1dace8b02d73693521 P fa551b23283b4034b256ba25acec541b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260925 03:02:53.675757 23508 raft_consensus.cc:1261] T 2baf62cd63564f4ba08b201e40e19d55 P 2edab657a33a4a0abfbd3262b1668483 [term 2 FOLLOWER]: Refusing update from remote peer fa551b23283b4034b256ba25acec541b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260925 03:02:53.677058 23665 consensus_queue.cc:1035] T 2baf62cd63564f4ba08b201e40e19d55 P fa551b23283b4034b256ba25acec541b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260925 03:02:53.685145 23582 raft_consensus.cc:1261] T 2baf62cd63564f4ba08b201e40e19d55 P b66c1d3fe3174f64b0936c512ee08f48 [term 2 FOLLOWER]: Refusing update from remote peer fa551b23283b4034b256ba25acec541b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260925 03:02:53.686669 23665 consensus_queue.cc:1035] T 2baf62cd63564f4ba08b201e40e19d55 P fa551b23283b4034b256ba25acec541b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260925 03:02:55.780635 23300 auto_leader_rebalancer-test.cc:269] The leader distribution is
I20260925 03:02:55.781411 23300 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260925 03:02:55.811765 23434 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2baf62cd63564f4ba08b201e40e19d55"
dest_uuid: "fa551b23283b4034b256ba25acec541b"
mode: GRACEFUL
new_leader_uuid: "b66c1d3fe3174f64b0936c512ee08f48"
from {username='slave'} at 127.0.0.1:35506
I20260925 03:02:55.812366 23434 raft_consensus.cc:595] T 2baf62cd63564f4ba08b201e40e19d55 P fa551b23283b4034b256ba25acec541b [term 2 LEADER]: Received request to transfer leadership to TS b66c1d3fe3174f64b0936c512ee08f48
I20260925 03:02:55.824040 23508 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c30e9a1092fe4d99994480aab8863cf6"
dest_uuid: "2edab657a33a4a0abfbd3262b1668483"
mode: GRACEFUL
new_leader_uuid: "b66c1d3fe3174f64b0936c512ee08f48"
from {username='slave'} at 127.0.0.1:45788
I20260925 03:02:55.824564 23508 raft_consensus.cc:595] T c30e9a1092fe4d99994480aab8863cf6 P 2edab657a33a4a0abfbd3262b1668483 [term 2 LEADER]: Received request to transfer leadership to TS b66c1d3fe3174f64b0936c512ee08f48
I20260925 03:02:55.825631 23300 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260925 03:02:55.826061 23300 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260925 03:02:55.896749 23723 raft_consensus.cc:979] T c30e9a1092fe4d99994480aab8863cf6 P 2edab657a33a4a0abfbd3262b1668483: : Instructing follower b66c1d3fe3174f64b0936c512ee08f48 to start an election
I20260925 03:02:55.897181 23695 raft_consensus.cc:1067] T c30e9a1092fe4d99994480aab8863cf6 P 2edab657a33a4a0abfbd3262b1668483 [term 2 LEADER]: Signalling peer b66c1d3fe3174f64b0936c512ee08f48 to start an election
I20260925 03:02:55.898437 23582 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c30e9a1092fe4d99994480aab8863cf6"
dest_uuid: "b66c1d3fe3174f64b0936c512ee08f48"
from {username='slave'} at 127.0.0.1:51168
I20260925 03:02:55.898859 23582 raft_consensus.cc:482] T c30e9a1092fe4d99994480aab8863cf6 P b66c1d3fe3174f64b0936c512ee08f48 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 03:02:55.899091 23582 raft_consensus.cc:3037] T c30e9a1092fe4d99994480aab8863cf6 P b66c1d3fe3174f64b0936c512ee08f48 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:02:55.903342 23582 raft_consensus.cc:504] T c30e9a1092fe4d99994480aab8863cf6 P b66c1d3fe3174f64b0936c512ee08f48 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:02:55.904801 23582 leader_election.cc:288] T c30e9a1092fe4d99994480aab8863cf6 P b66c1d3fe3174f64b0936c512ee08f48 [CANDIDATE]: Term 3 election: Requested vote from peers fa551b23283b4034b256ba25acec541b (127.22.193.1:42181), 2edab657a33a4a0abfbd3262b1668483 (127.22.193.2:35431)
I20260925 03:02:55.905594 23434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c30e9a1092fe4d99994480aab8863cf6" candidate_uuid: "b66c1d3fe3174f64b0936c512ee08f48" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "fa551b23283b4034b256ba25acec541b"
I20260925 03:02:55.905689 23508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c30e9a1092fe4d99994480aab8863cf6" candidate_uuid: "b66c1d3fe3174f64b0936c512ee08f48" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "2edab657a33a4a0abfbd3262b1668483"
I20260925 03:02:55.906077 23434 raft_consensus.cc:3037] T c30e9a1092fe4d99994480aab8863cf6 P fa551b23283b4034b256ba25acec541b [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:02:55.906186 23508 raft_consensus.cc:3032] T c30e9a1092fe4d99994480aab8863cf6 P 2edab657a33a4a0abfbd3262b1668483 [term 2 LEADER]: Stepping down as leader of term 2
I20260925 03:02:55.906553 23508 raft_consensus.cc:729] T c30e9a1092fe4d99994480aab8863cf6 P 2edab657a33a4a0abfbd3262b1668483 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2edab657a33a4a0abfbd3262b1668483, State: Running, Role: LEADER
I20260925 03:02:55.907147 23508 consensus_queue.cc:260] T c30e9a1092fe4d99994480aab8863cf6 P 2edab657a33a4a0abfbd3262b1668483 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:02:55.907996 23508 raft_consensus.cc:3037] T c30e9a1092fe4d99994480aab8863cf6 P 2edab657a33a4a0abfbd3262b1668483 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:02:55.910326 23434 raft_consensus.cc:2454] T c30e9a1092fe4d99994480aab8863cf6 P fa551b23283b4034b256ba25acec541b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b66c1d3fe3174f64b0936c512ee08f48 in term 3.
I20260925 03:02:55.911299 23547 leader_election.cc:302] T c30e9a1092fe4d99994480aab8863cf6 P b66c1d3fe3174f64b0936c512ee08f48 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b66c1d3fe3174f64b0936c512ee08f48, fa551b23283b4034b256ba25acec541b; no voters:
I20260925 03:02:55.911949 23722 raft_consensus.cc:2781] T c30e9a1092fe4d99994480aab8863cf6 P b66c1d3fe3174f64b0936c512ee08f48 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 03:02:55.913256 23722 raft_consensus.cc:686] T c30e9a1092fe4d99994480aab8863cf6 P b66c1d3fe3174f64b0936c512ee08f48 [term 3 LEADER]: Becoming Leader. State: Replica: b66c1d3fe3174f64b0936c512ee08f48, State: Running, Role: LEADER
I20260925 03:02:55.913525 23508 raft_consensus.cc:2454] T c30e9a1092fe4d99994480aab8863cf6 P 2edab657a33a4a0abfbd3262b1668483 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b66c1d3fe3174f64b0936c512ee08f48 in term 3.
I20260925 03:02:55.914039 23722 consensus_queue.cc:237] T c30e9a1092fe4d99994480aab8863cf6 P b66c1d3fe3174f64b0936c512ee08f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:02:55.922503 23328 catalog_manager.cc:5495] T c30e9a1092fe4d99994480aab8863cf6 P b66c1d3fe3174f64b0936c512ee08f48 reported cstate change: term changed from 2 to 3, leader changed from 2edab657a33a4a0abfbd3262b1668483 (127.22.193.2) to b66c1d3fe3174f64b0936c512ee08f48 (127.22.193.3). New cstate: current_term: 3 leader_uuid: "b66c1d3fe3174f64b0936c512ee08f48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } health_report { overall_health: HEALTHY } } }
I20260925 03:02:56.077369 23686 raft_consensus.cc:979] T 2baf62cd63564f4ba08b201e40e19d55 P fa551b23283b4034b256ba25acec541b: : Instructing follower b66c1d3fe3174f64b0936c512ee08f48 to start an election
I20260925 03:02:56.077798 23724 raft_consensus.cc:1067] T 2baf62cd63564f4ba08b201e40e19d55 P fa551b23283b4034b256ba25acec541b [term 2 LEADER]: Signalling peer b66c1d3fe3174f64b0936c512ee08f48 to start an election
I20260925 03:02:56.079218 23582 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2baf62cd63564f4ba08b201e40e19d55"
dest_uuid: "b66c1d3fe3174f64b0936c512ee08f48"
from {username='slave'} at 127.0.0.1:51184
I20260925 03:02:56.079712 23582 raft_consensus.cc:482] T 2baf62cd63564f4ba08b201e40e19d55 P b66c1d3fe3174f64b0936c512ee08f48 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 03:02:56.079982 23582 raft_consensus.cc:3037] T 2baf62cd63564f4ba08b201e40e19d55 P b66c1d3fe3174f64b0936c512ee08f48 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:02:56.085078 23582 raft_consensus.cc:504] T 2baf62cd63564f4ba08b201e40e19d55 P b66c1d3fe3174f64b0936c512ee08f48 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:02:56.086650 23582 leader_election.cc:288] T 2baf62cd63564f4ba08b201e40e19d55 P b66c1d3fe3174f64b0936c512ee08f48 [CANDIDATE]: Term 3 election: Requested vote from peers fa551b23283b4034b256ba25acec541b (127.22.193.1:42181), 2edab657a33a4a0abfbd3262b1668483 (127.22.193.2:35431)
I20260925 03:02:56.087479 23434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2baf62cd63564f4ba08b201e40e19d55" candidate_uuid: "b66c1d3fe3174f64b0936c512ee08f48" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "fa551b23283b4034b256ba25acec541b"
I20260925 03:02:56.087764 23508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2baf62cd63564f4ba08b201e40e19d55" candidate_uuid: "b66c1d3fe3174f64b0936c512ee08f48" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "2edab657a33a4a0abfbd3262b1668483"
I20260925 03:02:56.088110 23434 raft_consensus.cc:3032] T 2baf62cd63564f4ba08b201e40e19d55 P fa551b23283b4034b256ba25acec541b [term 2 LEADER]: Stepping down as leader of term 2
I20260925 03:02:56.088274 23508 raft_consensus.cc:3037] T 2baf62cd63564f4ba08b201e40e19d55 P 2edab657a33a4a0abfbd3262b1668483 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:02:56.088467 23434 raft_consensus.cc:729] T 2baf62cd63564f4ba08b201e40e19d55 P fa551b23283b4034b256ba25acec541b [term 2 LEADER]: Becoming Follower/Learner. State: Replica: fa551b23283b4034b256ba25acec541b, State: Running, Role: LEADER
I20260925 03:02:56.089344 23434 consensus_queue.cc:260] T 2baf62cd63564f4ba08b201e40e19d55 P fa551b23283b4034b256ba25acec541b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:02:56.090510 23434 raft_consensus.cc:3037] T 2baf62cd63564f4ba08b201e40e19d55 P fa551b23283b4034b256ba25acec541b [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:02:56.094888 23508 raft_consensus.cc:2454] T 2baf62cd63564f4ba08b201e40e19d55 P 2edab657a33a4a0abfbd3262b1668483 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b66c1d3fe3174f64b0936c512ee08f48 in term 3.
I20260925 03:02:56.096969 23548 leader_election.cc:302] T 2baf62cd63564f4ba08b201e40e19d55 P b66c1d3fe3174f64b0936c512ee08f48 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2edab657a33a4a0abfbd3262b1668483, b66c1d3fe3174f64b0936c512ee08f48; no voters:
I20260925 03:02:56.097388 23434 raft_consensus.cc:2454] T 2baf62cd63564f4ba08b201e40e19d55 P fa551b23283b4034b256ba25acec541b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b66c1d3fe3174f64b0936c512ee08f48 in term 3.
I20260925 03:02:56.097756 23722 raft_consensus.cc:2781] T 2baf62cd63564f4ba08b201e40e19d55 P b66c1d3fe3174f64b0936c512ee08f48 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 03:02:56.098235 23722 raft_consensus.cc:686] T 2baf62cd63564f4ba08b201e40e19d55 P b66c1d3fe3174f64b0936c512ee08f48 [term 3 LEADER]: Becoming Leader. State: Replica: b66c1d3fe3174f64b0936c512ee08f48, State: Running, Role: LEADER
I20260925 03:02:56.099061 23722 consensus_queue.cc:237] T 2baf62cd63564f4ba08b201e40e19d55 P b66c1d3fe3174f64b0936c512ee08f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:02:56.107749 23332 catalog_manager.cc:5495] T 2baf62cd63564f4ba08b201e40e19d55 P b66c1d3fe3174f64b0936c512ee08f48 reported cstate change: term changed from 2 to 3, leader changed from fa551b23283b4034b256ba25acec541b (127.22.193.1) to b66c1d3fe3174f64b0936c512ee08f48 (127.22.193.3). New cstate: current_term: 3 leader_uuid: "b66c1d3fe3174f64b0936c512ee08f48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } health_report { overall_health: HEALTHY } } }
I20260925 03:02:56.332372 23508 raft_consensus.cc:1261] T c30e9a1092fe4d99994480aab8863cf6 P 2edab657a33a4a0abfbd3262b1668483 [term 3 FOLLOWER]: Refusing update from remote peer b66c1d3fe3174f64b0936c512ee08f48: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260925 03:02:56.333730 23722 consensus_queue.cc:1035] T c30e9a1092fe4d99994480aab8863cf6 P b66c1d3fe3174f64b0936c512ee08f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260925 03:02:56.342887 23434 raft_consensus.cc:1261] T c30e9a1092fe4d99994480aab8863cf6 P fa551b23283b4034b256ba25acec541b [term 3 FOLLOWER]: Refusing update from remote peer b66c1d3fe3174f64b0936c512ee08f48: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260925 03:02:56.344332 23722 consensus_queue.cc:1035] T c30e9a1092fe4d99994480aab8863cf6 P b66c1d3fe3174f64b0936c512ee08f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260925 03:02:56.525512 23508 raft_consensus.cc:1261] T 2baf62cd63564f4ba08b201e40e19d55 P 2edab657a33a4a0abfbd3262b1668483 [term 3 FOLLOWER]: Refusing update from remote peer b66c1d3fe3174f64b0936c512ee08f48: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260925 03:02:56.526983 23722 consensus_queue.cc:1035] T 2baf62cd63564f4ba08b201e40e19d55 P b66c1d3fe3174f64b0936c512ee08f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260925 03:02:56.536017 23434 raft_consensus.cc:1261] T 2baf62cd63564f4ba08b201e40e19d55 P fa551b23283b4034b256ba25acec541b [term 3 FOLLOWER]: Refusing update from remote peer b66c1d3fe3174f64b0936c512ee08f48: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260925 03:02:56.537437 23722 consensus_queue.cc:1035] T 2baf62cd63564f4ba08b201e40e19d55 P b66c1d3fe3174f64b0936c512ee08f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260925 03:02:56.827000 23300 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260925 03:02:56.831969 23300 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260925 03:02:56.832386 23300 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260925 03:02:57.833185 23300 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260925 03:02:57.838343 23300 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260925 03:02:57.838733 23300 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260925 03:02:58.839784 23300 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260925 03:02:58.845170 23300 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260925 03:02:58.845616 23300 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260925 03:02:59.846443 23300 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260925 03:02:59.851203 23300 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260925 03:02:59.851588 23300 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260925 03:03:00.852383 23300 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260925 03:03:00.857144 23300 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260925 03:03:00.857549 23300 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260925 03:03:01.858300 23300 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260925 03:03:01.862907 23300 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260925 03:03:01.863330 23300 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260925 03:03:02.864197 23300 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260925 03:03:02.869282 23300 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260925 03:03:02.869758 23300 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260925 03:03:03.870803 23300 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260925 03:03:03.875674 23300 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260925 03:03:03.876078 23300 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260925 03:03:04.876876 23300 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260925 03:03:04.881548 23300 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260925 03:03:04.881968 23300 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260925 03:03:05.910171 23300 auto_leader_rebalancer-test.cc:288] The leader distribution is
I20260925 03:03:05.915064 23434 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ef74cafe53e34e1dace8b02d73693521"
dest_uuid: "fa551b23283b4034b256ba25acec541b"
mode: GRACEFUL
new_leader_uuid: "2edab657a33a4a0abfbd3262b1668483"
from {username='slave'} at 127.0.0.1:40566
I20260925 03:03:05.915831 23434 raft_consensus.cc:595] T ef74cafe53e34e1dace8b02d73693521 P fa551b23283b4034b256ba25acec541b [term 2 LEADER]: Received request to transfer leadership to TS 2edab657a33a4a0abfbd3262b1668483
I20260925 03:03:05.920616 23434 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d01220d66a8c4d34ac86fc3559fa5165"
dest_uuid: "fa551b23283b4034b256ba25acec541b"
mode: GRACEFUL
new_leader_uuid: "2edab657a33a4a0abfbd3262b1668483"
from {username='slave'} at 127.0.0.1:40566
I20260925 03:03:05.921097 23434 raft_consensus.cc:595] T d01220d66a8c4d34ac86fc3559fa5165 P fa551b23283b4034b256ba25acec541b [term 2 LEADER]: Received request to transfer leadership to TS 2edab657a33a4a0abfbd3262b1668483
I20260925 03:03:05.925441 23434 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "832e06098a3e499f82d53853e26235cf"
dest_uuid: "fa551b23283b4034b256ba25acec541b"
mode: GRACEFUL
new_leader_uuid: "2edab657a33a4a0abfbd3262b1668483"
from {username='slave'} at 127.0.0.1:40566
I20260925 03:03:05.925907 23434 raft_consensus.cc:595] T 832e06098a3e499f82d53853e26235cf P fa551b23283b4034b256ba25acec541b [term 2 LEADER]: Received request to transfer leadership to TS 2edab657a33a4a0abfbd3262b1668483
I20260925 03:03:05.930635 23582 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2baf62cd63564f4ba08b201e40e19d55"
dest_uuid: "b66c1d3fe3174f64b0936c512ee08f48"
mode: GRACEFUL
new_leader_uuid: "2edab657a33a4a0abfbd3262b1668483"
from {username='slave'} at 127.0.0.1:51192
I20260925 03:03:05.931192 23582 raft_consensus.cc:595] T 2baf62cd63564f4ba08b201e40e19d55 P b66c1d3fe3174f64b0936c512ee08f48 [term 3 LEADER]: Received request to transfer leadership to TS 2edab657a33a4a0abfbd3262b1668483
I20260925 03:03:05.933440 23748 raft_consensus.cc:979] T 2baf62cd63564f4ba08b201e40e19d55 P b66c1d3fe3174f64b0936c512ee08f48: : Instructing follower 2edab657a33a4a0abfbd3262b1668483 to start an election
I20260925 03:03:05.934052 23728 raft_consensus.cc:1067] T 2baf62cd63564f4ba08b201e40e19d55 P b66c1d3fe3174f64b0936c512ee08f48 [term 3 LEADER]: Signalling peer 2edab657a33a4a0abfbd3262b1668483 to start an election
I20260925 03:03:05.935560 23508 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2baf62cd63564f4ba08b201e40e19d55"
dest_uuid: "2edab657a33a4a0abfbd3262b1668483"
from {username='slave'} at 127.0.0.1:38970
I20260925 03:03:05.936090 23508 raft_consensus.cc:482] T 2baf62cd63564f4ba08b201e40e19d55 P 2edab657a33a4a0abfbd3262b1668483 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 03:03:05.936510 23508 raft_consensus.cc:3037] T 2baf62cd63564f4ba08b201e40e19d55 P 2edab657a33a4a0abfbd3262b1668483 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:03:05.936592 23582 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c30e9a1092fe4d99994480aab8863cf6"
dest_uuid: "b66c1d3fe3174f64b0936c512ee08f48"
mode: GRACEFUL
new_leader_uuid: "2edab657a33a4a0abfbd3262b1668483"
from {username='slave'} at 127.0.0.1:51192
I20260925 03:03:05.937163 23582 raft_consensus.cc:595] T c30e9a1092fe4d99994480aab8863cf6 P b66c1d3fe3174f64b0936c512ee08f48 [term 3 LEADER]: Received request to transfer leadership to TS 2edab657a33a4a0abfbd3262b1668483
I20260925 03:03:05.943063 23508 raft_consensus.cc:504] T 2baf62cd63564f4ba08b201e40e19d55 P 2edab657a33a4a0abfbd3262b1668483 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:03:05.945343 23508 leader_election.cc:288] T 2baf62cd63564f4ba08b201e40e19d55 P 2edab657a33a4a0abfbd3262b1668483 [CANDIDATE]: Term 4 election: Requested vote from peers fa551b23283b4034b256ba25acec541b (127.22.193.1:42181), b66c1d3fe3174f64b0936c512ee08f48 (127.22.193.3:42621)
I20260925 03:03:05.946064 23434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2baf62cd63564f4ba08b201e40e19d55" candidate_uuid: "2edab657a33a4a0abfbd3262b1668483" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "fa551b23283b4034b256ba25acec541b"
I20260925 03:03:05.946286 23582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2baf62cd63564f4ba08b201e40e19d55" candidate_uuid: "2edab657a33a4a0abfbd3262b1668483" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "b66c1d3fe3174f64b0936c512ee08f48"
I20260925 03:03:05.946959 23582 raft_consensus.cc:3032] T 2baf62cd63564f4ba08b201e40e19d55 P b66c1d3fe3174f64b0936c512ee08f48 [term 3 LEADER]: Stepping down as leader of term 3
I20260925 03:03:05.946890 23434 raft_consensus.cc:3037] T 2baf62cd63564f4ba08b201e40e19d55 P fa551b23283b4034b256ba25acec541b [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:03:05.947327 23582 raft_consensus.cc:729] T 2baf62cd63564f4ba08b201e40e19d55 P b66c1d3fe3174f64b0936c512ee08f48 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: b66c1d3fe3174f64b0936c512ee08f48, State: Running, Role: LEADER
I20260925 03:03:05.948055 23582 consensus_queue.cc:260] T 2baf62cd63564f4ba08b201e40e19d55 P b66c1d3fe3174f64b0936c512ee08f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:03:05.949276 23582 raft_consensus.cc:3037] T 2baf62cd63564f4ba08b201e40e19d55 P b66c1d3fe3174f64b0936c512ee08f48 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:03:05.953008 23434 raft_consensus.cc:2454] T 2baf62cd63564f4ba08b201e40e19d55 P fa551b23283b4034b256ba25acec541b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2edab657a33a4a0abfbd3262b1668483 in term 4.
I20260925 03:03:05.954089 23473 leader_election.cc:302] T 2baf62cd63564f4ba08b201e40e19d55 P 2edab657a33a4a0abfbd3262b1668483 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2edab657a33a4a0abfbd3262b1668483, fa551b23283b4034b256ba25acec541b; no voters:
I20260925 03:03:05.954469 23582 raft_consensus.cc:2454] T 2baf62cd63564f4ba08b201e40e19d55 P b66c1d3fe3174f64b0936c512ee08f48 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2edab657a33a4a0abfbd3262b1668483 in term 4.
I20260925 03:03:05.955065 23745 raft_consensus.cc:2781] T 2baf62cd63564f4ba08b201e40e19d55 P 2edab657a33a4a0abfbd3262b1668483 [term 4 FOLLOWER]: Leader election won for term 4
I20260925 03:03:05.956280 23745 raft_consensus.cc:686] T 2baf62cd63564f4ba08b201e40e19d55 P 2edab657a33a4a0abfbd3262b1668483 [term 4 LEADER]: Becoming Leader. State: Replica: 2edab657a33a4a0abfbd3262b1668483, State: Running, Role: LEADER
I20260925 03:03:05.957134 23745 consensus_queue.cc:237] T 2baf62cd63564f4ba08b201e40e19d55 P 2edab657a33a4a0abfbd3262b1668483 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:03:05.964371 23331 catalog_manager.cc:5495] T 2baf62cd63564f4ba08b201e40e19d55 P 2edab657a33a4a0abfbd3262b1668483 reported cstate change: term changed from 3 to 4, leader changed from b66c1d3fe3174f64b0936c512ee08f48 (127.22.193.3) to 2edab657a33a4a0abfbd3262b1668483 (127.22.193.2). New cstate: current_term: 4 leader_uuid: "2edab657a33a4a0abfbd3262b1668483" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } health_report { overall_health: UNKNOWN } } }
I20260925 03:03:06.028328 23724 raft_consensus.cc:979] T d01220d66a8c4d34ac86fc3559fa5165 P fa551b23283b4034b256ba25acec541b: : Instructing follower 2edab657a33a4a0abfbd3262b1668483 to start an election
I20260925 03:03:06.028825 23755 raft_consensus.cc:1067] T d01220d66a8c4d34ac86fc3559fa5165 P fa551b23283b4034b256ba25acec541b [term 2 LEADER]: Signalling peer 2edab657a33a4a0abfbd3262b1668483 to start an election
I20260925 03:03:06.030090 23508 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d01220d66a8c4d34ac86fc3559fa5165"
dest_uuid: "2edab657a33a4a0abfbd3262b1668483"
from {username='slave'} at 127.0.0.1:38974
I20260925 03:03:06.030539 23508 raft_consensus.cc:482] T d01220d66a8c4d34ac86fc3559fa5165 P 2edab657a33a4a0abfbd3262b1668483 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 03:03:06.030825 23508 raft_consensus.cc:3037] T d01220d66a8c4d34ac86fc3559fa5165 P 2edab657a33a4a0abfbd3262b1668483 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:03:06.035233 23508 raft_consensus.cc:504] T d01220d66a8c4d34ac86fc3559fa5165 P 2edab657a33a4a0abfbd3262b1668483 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:03:06.036612 23508 leader_election.cc:288] T d01220d66a8c4d34ac86fc3559fa5165 P 2edab657a33a4a0abfbd3262b1668483 [CANDIDATE]: Term 3 election: Requested vote from peers fa551b23283b4034b256ba25acec541b (127.22.193.1:42181), b66c1d3fe3174f64b0936c512ee08f48 (127.22.193.3:42621)
I20260925 03:03:06.037477 23434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d01220d66a8c4d34ac86fc3559fa5165" candidate_uuid: "2edab657a33a4a0abfbd3262b1668483" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "fa551b23283b4034b256ba25acec541b"
I20260925 03:03:06.037619 23582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d01220d66a8c4d34ac86fc3559fa5165" candidate_uuid: "2edab657a33a4a0abfbd3262b1668483" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "b66c1d3fe3174f64b0936c512ee08f48"
I20260925 03:03:06.038054 23434 raft_consensus.cc:3032] T d01220d66a8c4d34ac86fc3559fa5165 P fa551b23283b4034b256ba25acec541b [term 2 LEADER]: Stepping down as leader of term 2
I20260925 03:03:06.038090 23582 raft_consensus.cc:3037] T d01220d66a8c4d34ac86fc3559fa5165 P b66c1d3fe3174f64b0936c512ee08f48 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:03:06.038350 23434 raft_consensus.cc:729] T d01220d66a8c4d34ac86fc3559fa5165 P fa551b23283b4034b256ba25acec541b [term 2 LEADER]: Becoming Follower/Learner. State: Replica: fa551b23283b4034b256ba25acec541b, State: Running, Role: LEADER
I20260925 03:03:06.038988 23434 consensus_queue.cc:260] T d01220d66a8c4d34ac86fc3559fa5165 P fa551b23283b4034b256ba25acec541b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:03:06.039929 23434 raft_consensus.cc:3037] T d01220d66a8c4d34ac86fc3559fa5165 P fa551b23283b4034b256ba25acec541b [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:03:06.042498 23582 raft_consensus.cc:2454] T d01220d66a8c4d34ac86fc3559fa5165 P b66c1d3fe3174f64b0936c512ee08f48 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2edab657a33a4a0abfbd3262b1668483 in term 3.
I20260925 03:03:06.043406 23473 leader_election.cc:302] T d01220d66a8c4d34ac86fc3559fa5165 P 2edab657a33a4a0abfbd3262b1668483 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2edab657a33a4a0abfbd3262b1668483, b66c1d3fe3174f64b0936c512ee08f48; no voters:
I20260925 03:03:06.044122 23434 raft_consensus.cc:2454] T d01220d66a8c4d34ac86fc3559fa5165 P fa551b23283b4034b256ba25acec541b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2edab657a33a4a0abfbd3262b1668483 in term 3.
I20260925 03:03:06.044164 23745 raft_consensus.cc:2781] T d01220d66a8c4d34ac86fc3559fa5165 P 2edab657a33a4a0abfbd3262b1668483 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 03:03:06.044615 23745 raft_consensus.cc:686] T d01220d66a8c4d34ac86fc3559fa5165 P 2edab657a33a4a0abfbd3262b1668483 [term 3 LEADER]: Becoming Leader. State: Replica: 2edab657a33a4a0abfbd3262b1668483, State: Running, Role: LEADER
I20260925 03:03:06.045442 23745 consensus_queue.cc:237] T d01220d66a8c4d34ac86fc3559fa5165 P 2edab657a33a4a0abfbd3262b1668483 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:03:06.052438 23331 catalog_manager.cc:5495] T d01220d66a8c4d34ac86fc3559fa5165 P 2edab657a33a4a0abfbd3262b1668483 reported cstate change: term changed from 2 to 3, leader changed from fa551b23283b4034b256ba25acec541b (127.22.193.1) to 2edab657a33a4a0abfbd3262b1668483 (127.22.193.2). New cstate: current_term: 3 leader_uuid: "2edab657a33a4a0abfbd3262b1668483" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } health_report { overall_health: UNKNOWN } } }
I20260925 03:03:06.065094 23724 raft_consensus.cc:979] T ef74cafe53e34e1dace8b02d73693521 P fa551b23283b4034b256ba25acec541b: : Instructing follower 2edab657a33a4a0abfbd3262b1668483 to start an election
I20260925 03:03:06.065487 23755 raft_consensus.cc:1067] T ef74cafe53e34e1dace8b02d73693521 P fa551b23283b4034b256ba25acec541b [term 2 LEADER]: Signalling peer 2edab657a33a4a0abfbd3262b1668483 to start an election
I20260925 03:03:06.066681 23508 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ef74cafe53e34e1dace8b02d73693521"
dest_uuid: "2edab657a33a4a0abfbd3262b1668483"
from {username='slave'} at 127.0.0.1:38974
I20260925 03:03:06.067117 23508 raft_consensus.cc:482] T ef74cafe53e34e1dace8b02d73693521 P 2edab657a33a4a0abfbd3262b1668483 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 03:03:06.067389 23508 raft_consensus.cc:3037] T ef74cafe53e34e1dace8b02d73693521 P 2edab657a33a4a0abfbd3262b1668483 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:03:06.071941 23508 raft_consensus.cc:504] T ef74cafe53e34e1dace8b02d73693521 P 2edab657a33a4a0abfbd3262b1668483 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } } peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } }
I20260925 03:03:06.073489 23508 leader_election.cc:288] T ef74cafe53e34e1dace8b02d73693521 P 2edab657a33a4a0abfbd3262b1668483 [CANDIDATE]: Term 3 election: Requested vote from peers b66c1d3fe3174f64b0936c512ee08f48 (127.22.193.3:42621), fa551b23283b4034b256ba25acec541b (127.22.193.1:42181)
I20260925 03:03:06.074195 23582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef74cafe53e34e1dace8b02d73693521" candidate_uuid: "2edab657a33a4a0abfbd3262b1668483" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "b66c1d3fe3174f64b0936c512ee08f48"
I20260925 03:03:06.074378 23434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef74cafe53e34e1dace8b02d73693521" candidate_uuid: "2edab657a33a4a0abfbd3262b1668483" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "fa551b23283b4034b256ba25acec541b"
I20260925 03:03:06.074746 23582 raft_consensus.cc:3037] T ef74cafe53e34e1dace8b02d73693521 P b66c1d3fe3174f64b0936c512ee08f48 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:03:06.074819 23434 raft_consensus.cc:3032] T ef74cafe53e34e1dace8b02d73693521 P fa551b23283b4034b256ba25acec541b [term 2 LEADER]: Stepping down as leader of term 2
I20260925 03:03:06.075098 23434 raft_consensus.cc:729] T ef74cafe53e34e1dace8b02d73693521 P fa551b23283b4034b256ba25acec541b [term 2 LEADER]: Becoming Follower/Learner. State: Replica: fa551b23283b4034b256ba25acec541b, State: Running, Role: LEADER
I20260925 03:03:06.075631 23434 consensus_queue.cc:260] T ef74cafe53e34e1dace8b02d73693521 P fa551b23283b4034b256ba25acec541b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } } peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } }
I20260925 03:03:06.076449 23434 raft_consensus.cc:3037] T ef74cafe53e34e1dace8b02d73693521 P fa551b23283b4034b256ba25acec541b [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:03:06.078997 23582 raft_consensus.cc:2454] T ef74cafe53e34e1dace8b02d73693521 P b66c1d3fe3174f64b0936c512ee08f48 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2edab657a33a4a0abfbd3262b1668483 in term 3.
I20260925 03:03:06.079910 23473 leader_election.cc:302] T ef74cafe53e34e1dace8b02d73693521 P 2edab657a33a4a0abfbd3262b1668483 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2edab657a33a4a0abfbd3262b1668483, b66c1d3fe3174f64b0936c512ee08f48; no voters:
I20260925 03:03:06.080631 23434 raft_consensus.cc:2454] T ef74cafe53e34e1dace8b02d73693521 P fa551b23283b4034b256ba25acec541b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2edab657a33a4a0abfbd3262b1668483 in term 3.
I20260925 03:03:06.080641 23745 raft_consensus.cc:2781] T ef74cafe53e34e1dace8b02d73693521 P 2edab657a33a4a0abfbd3262b1668483 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 03:03:06.081233 23745 raft_consensus.cc:686] T ef74cafe53e34e1dace8b02d73693521 P 2edab657a33a4a0abfbd3262b1668483 [term 3 LEADER]: Becoming Leader. State: Replica: 2edab657a33a4a0abfbd3262b1668483, State: Running, Role: LEADER
I20260925 03:03:06.081992 23745 consensus_queue.cc:237] T ef74cafe53e34e1dace8b02d73693521 P 2edab657a33a4a0abfbd3262b1668483 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } } peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } }
I20260925 03:03:06.088208 23331 catalog_manager.cc:5495] T ef74cafe53e34e1dace8b02d73693521 P 2edab657a33a4a0abfbd3262b1668483 reported cstate change: term changed from 2 to 3, leader changed from fa551b23283b4034b256ba25acec541b (127.22.193.1) to 2edab657a33a4a0abfbd3262b1668483 (127.22.193.2). New cstate: current_term: 3 leader_uuid: "2edab657a33a4a0abfbd3262b1668483" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } health_report { overall_health: HEALTHY } } }
I20260925 03:03:06.234130 23748 raft_consensus.cc:979] T c30e9a1092fe4d99994480aab8863cf6 P b66c1d3fe3174f64b0936c512ee08f48: : Instructing follower 2edab657a33a4a0abfbd3262b1668483 to start an election
I20260925 03:03:06.234627 23728 raft_consensus.cc:1067] T c30e9a1092fe4d99994480aab8863cf6 P b66c1d3fe3174f64b0936c512ee08f48 [term 3 LEADER]: Signalling peer 2edab657a33a4a0abfbd3262b1668483 to start an election
I20260925 03:03:06.235937 23508 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c30e9a1092fe4d99994480aab8863cf6"
dest_uuid: "2edab657a33a4a0abfbd3262b1668483"
from {username='slave'} at 127.0.0.1:38970
I20260925 03:03:06.236369 23508 raft_consensus.cc:482] T c30e9a1092fe4d99994480aab8863cf6 P 2edab657a33a4a0abfbd3262b1668483 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 03:03:06.236611 23508 raft_consensus.cc:3037] T c30e9a1092fe4d99994480aab8863cf6 P 2edab657a33a4a0abfbd3262b1668483 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:03:06.240895 23508 raft_consensus.cc:504] T c30e9a1092fe4d99994480aab8863cf6 P 2edab657a33a4a0abfbd3262b1668483 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:03:06.242254 23508 leader_election.cc:288] T c30e9a1092fe4d99994480aab8863cf6 P 2edab657a33a4a0abfbd3262b1668483 [CANDIDATE]: Term 4 election: Requested vote from peers fa551b23283b4034b256ba25acec541b (127.22.193.1:42181), b66c1d3fe3174f64b0936c512ee08f48 (127.22.193.3:42621)
I20260925 03:03:06.243163 23434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c30e9a1092fe4d99994480aab8863cf6" candidate_uuid: "2edab657a33a4a0abfbd3262b1668483" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "fa551b23283b4034b256ba25acec541b"
I20260925 03:03:06.243268 23582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c30e9a1092fe4d99994480aab8863cf6" candidate_uuid: "2edab657a33a4a0abfbd3262b1668483" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "b66c1d3fe3174f64b0936c512ee08f48"
I20260925 03:03:06.243660 23434 raft_consensus.cc:3037] T c30e9a1092fe4d99994480aab8863cf6 P fa551b23283b4034b256ba25acec541b [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:03:06.243753 23582 raft_consensus.cc:3032] T c30e9a1092fe4d99994480aab8863cf6 P b66c1d3fe3174f64b0936c512ee08f48 [term 3 LEADER]: Stepping down as leader of term 3
I20260925 03:03:06.244016 23582 raft_consensus.cc:729] T c30e9a1092fe4d99994480aab8863cf6 P b66c1d3fe3174f64b0936c512ee08f48 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: b66c1d3fe3174f64b0936c512ee08f48, State: Running, Role: LEADER
I20260925 03:03:06.244549 23582 consensus_queue.cc:260] T c30e9a1092fe4d99994480aab8863cf6 P b66c1d3fe3174f64b0936c512ee08f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:03:06.245406 23582 raft_consensus.cc:3037] T c30e9a1092fe4d99994480aab8863cf6 P b66c1d3fe3174f64b0936c512ee08f48 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:03:06.248287 23434 raft_consensus.cc:2454] T c30e9a1092fe4d99994480aab8863cf6 P fa551b23283b4034b256ba25acec541b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2edab657a33a4a0abfbd3262b1668483 in term 4.
I20260925 03:03:06.249279 23473 leader_election.cc:302] T c30e9a1092fe4d99994480aab8863cf6 P 2edab657a33a4a0abfbd3262b1668483 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2edab657a33a4a0abfbd3262b1668483, fa551b23283b4034b256ba25acec541b; no voters:
I20260925 03:03:06.249538 23582 raft_consensus.cc:2454] T c30e9a1092fe4d99994480aab8863cf6 P b66c1d3fe3174f64b0936c512ee08f48 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2edab657a33a4a0abfbd3262b1668483 in term 4.
I20260925 03:03:06.249922 23745 raft_consensus.cc:2781] T c30e9a1092fe4d99994480aab8863cf6 P 2edab657a33a4a0abfbd3262b1668483 [term 4 FOLLOWER]: Leader election won for term 4
I20260925 03:03:06.250365 23745 raft_consensus.cc:686] T c30e9a1092fe4d99994480aab8863cf6 P 2edab657a33a4a0abfbd3262b1668483 [term 4 LEADER]: Becoming Leader. State: Replica: 2edab657a33a4a0abfbd3262b1668483, State: Running, Role: LEADER
I20260925 03:03:06.251149 23745 consensus_queue.cc:237] T c30e9a1092fe4d99994480aab8863cf6 P 2edab657a33a4a0abfbd3262b1668483 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:03:06.257424 23331 catalog_manager.cc:5495] T c30e9a1092fe4d99994480aab8863cf6 P 2edab657a33a4a0abfbd3262b1668483 reported cstate change: term changed from 3 to 4, leader changed from b66c1d3fe3174f64b0936c512ee08f48 (127.22.193.3) to 2edab657a33a4a0abfbd3262b1668483 (127.22.193.2). New cstate: current_term: 4 leader_uuid: "2edab657a33a4a0abfbd3262b1668483" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } health_report { overall_health: UNKNOWN } } }
I20260925 03:03:06.264298 23724 raft_consensus.cc:979] T 832e06098a3e499f82d53853e26235cf P fa551b23283b4034b256ba25acec541b: : Instructing follower 2edab657a33a4a0abfbd3262b1668483 to start an election
I20260925 03:03:06.264734 23755 raft_consensus.cc:1067] T 832e06098a3e499f82d53853e26235cf P fa551b23283b4034b256ba25acec541b [term 2 LEADER]: Signalling peer 2edab657a33a4a0abfbd3262b1668483 to start an election
I20260925 03:03:06.266130 23508 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "832e06098a3e499f82d53853e26235cf"
dest_uuid: "2edab657a33a4a0abfbd3262b1668483"
from {username='slave'} at 127.0.0.1:38974
I20260925 03:03:06.266647 23508 raft_consensus.cc:482] T 832e06098a3e499f82d53853e26235cf P 2edab657a33a4a0abfbd3262b1668483 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 03:03:06.266964 23508 raft_consensus.cc:3037] T 832e06098a3e499f82d53853e26235cf P 2edab657a33a4a0abfbd3262b1668483 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:03:06.271140 23508 raft_consensus.cc:504] T 832e06098a3e499f82d53853e26235cf P 2edab657a33a4a0abfbd3262b1668483 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:03:06.272414 23508 leader_election.cc:288] T 832e06098a3e499f82d53853e26235cf P 2edab657a33a4a0abfbd3262b1668483 [CANDIDATE]: Term 3 election: Requested vote from peers fa551b23283b4034b256ba25acec541b (127.22.193.1:42181), b66c1d3fe3174f64b0936c512ee08f48 (127.22.193.3:42621)
I20260925 03:03:06.273332 23434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "832e06098a3e499f82d53853e26235cf" candidate_uuid: "2edab657a33a4a0abfbd3262b1668483" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "fa551b23283b4034b256ba25acec541b"
I20260925 03:03:06.273453 23582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "832e06098a3e499f82d53853e26235cf" candidate_uuid: "2edab657a33a4a0abfbd3262b1668483" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "b66c1d3fe3174f64b0936c512ee08f48"
I20260925 03:03:06.273949 23434 raft_consensus.cc:3032] T 832e06098a3e499f82d53853e26235cf P fa551b23283b4034b256ba25acec541b [term 2 LEADER]: Stepping down as leader of term 2
I20260925 03:03:06.273979 23582 raft_consensus.cc:3037] T 832e06098a3e499f82d53853e26235cf P b66c1d3fe3174f64b0936c512ee08f48 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:03:06.274365 23434 raft_consensus.cc:729] T 832e06098a3e499f82d53853e26235cf P fa551b23283b4034b256ba25acec541b [term 2 LEADER]: Becoming Follower/Learner. State: Replica: fa551b23283b4034b256ba25acec541b, State: Running, Role: LEADER
I20260925 03:03:06.275033 23434 consensus_queue.cc:260] T 832e06098a3e499f82d53853e26235cf P fa551b23283b4034b256ba25acec541b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:03:06.276161 23434 raft_consensus.cc:3037] T 832e06098a3e499f82d53853e26235cf P fa551b23283b4034b256ba25acec541b [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:03:06.280205 23582 raft_consensus.cc:2454] T 832e06098a3e499f82d53853e26235cf P b66c1d3fe3174f64b0936c512ee08f48 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2edab657a33a4a0abfbd3262b1668483 in term 3.
I20260925 03:03:06.281255 23473 leader_election.cc:302] T 832e06098a3e499f82d53853e26235cf P 2edab657a33a4a0abfbd3262b1668483 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2edab657a33a4a0abfbd3262b1668483, b66c1d3fe3174f64b0936c512ee08f48; no voters:
I20260925 03:03:06.281975 23434 raft_consensus.cc:2454] T 832e06098a3e499f82d53853e26235cf P fa551b23283b4034b256ba25acec541b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2edab657a33a4a0abfbd3262b1668483 in term 3.
I20260925 03:03:06.281998 23745 raft_consensus.cc:2781] T 832e06098a3e499f82d53853e26235cf P 2edab657a33a4a0abfbd3262b1668483 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 03:03:06.282507 23745 raft_consensus.cc:686] T 832e06098a3e499f82d53853e26235cf P 2edab657a33a4a0abfbd3262b1668483 [term 3 LEADER]: Becoming Leader. State: Replica: 2edab657a33a4a0abfbd3262b1668483, State: Running, Role: LEADER
I20260925 03:03:06.283275 23745 consensus_queue.cc:237] T 832e06098a3e499f82d53853e26235cf P 2edab657a33a4a0abfbd3262b1668483 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:03:06.289888 23331 catalog_manager.cc:5495] T 832e06098a3e499f82d53853e26235cf P 2edab657a33a4a0abfbd3262b1668483 reported cstate change: term changed from 2 to 3, leader changed from fa551b23283b4034b256ba25acec541b (127.22.193.1) to 2edab657a33a4a0abfbd3262b1668483 (127.22.193.2). New cstate: current_term: 3 leader_uuid: "2edab657a33a4a0abfbd3262b1668483" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } health_report { overall_health: UNKNOWN } } }
I20260925 03:03:06.481405 23582 raft_consensus.cc:1261] T ef74cafe53e34e1dace8b02d73693521 P b66c1d3fe3174f64b0936c512ee08f48 [term 3 FOLLOWER]: Refusing update from remote peer 2edab657a33a4a0abfbd3262b1668483: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260925 03:03:06.482504 23745 consensus_queue.cc:1035] T ef74cafe53e34e1dace8b02d73693521 P 2edab657a33a4a0abfbd3262b1668483 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260925 03:03:06.492410 23434 raft_consensus.cc:1261] T ef74cafe53e34e1dace8b02d73693521 P fa551b23283b4034b256ba25acec541b [term 3 FOLLOWER]: Refusing update from remote peer 2edab657a33a4a0abfbd3262b1668483: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260925 03:03:06.493845 23763 consensus_queue.cc:1035] T ef74cafe53e34e1dace8b02d73693521 P 2edab657a33a4a0abfbd3262b1668483 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260925 03:03:06.512903 23582 raft_consensus.cc:1261] T 2baf62cd63564f4ba08b201e40e19d55 P b66c1d3fe3174f64b0936c512ee08f48 [term 4 FOLLOWER]: Refusing update from remote peer 2edab657a33a4a0abfbd3262b1668483: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260925 03:03:06.513924 23763 consensus_queue.cc:1035] T 2baf62cd63564f4ba08b201e40e19d55 P 2edab657a33a4a0abfbd3262b1668483 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260925 03:03:06.521350 23582 raft_consensus.cc:1261] T d01220d66a8c4d34ac86fc3559fa5165 P b66c1d3fe3174f64b0936c512ee08f48 [term 3 FOLLOWER]: Refusing update from remote peer 2edab657a33a4a0abfbd3262b1668483: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260925 03:03:06.523262 23749 consensus_queue.cc:1035] T d01220d66a8c4d34ac86fc3559fa5165 P 2edab657a33a4a0abfbd3262b1668483 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260925 03:03:06.524264 23434 raft_consensus.cc:1261] T 2baf62cd63564f4ba08b201e40e19d55 P fa551b23283b4034b256ba25acec541b [term 4 FOLLOWER]: Refusing update from remote peer 2edab657a33a4a0abfbd3262b1668483: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260925 03:03:06.526875 23763 consensus_queue.cc:1035] T 2baf62cd63564f4ba08b201e40e19d55 P 2edab657a33a4a0abfbd3262b1668483 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260925 03:03:06.538900 23434 raft_consensus.cc:1261] T d01220d66a8c4d34ac86fc3559fa5165 P fa551b23283b4034b256ba25acec541b [term 3 FOLLOWER]: Refusing update from remote peer 2edab657a33a4a0abfbd3262b1668483: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260925 03:03:06.540289 23749 consensus_queue.cc:1035] T d01220d66a8c4d34ac86fc3559fa5165 P 2edab657a33a4a0abfbd3262b1668483 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260925 03:03:06.682104 23434 raft_consensus.cc:1261] T c30e9a1092fe4d99994480aab8863cf6 P fa551b23283b4034b256ba25acec541b [term 4 FOLLOWER]: Refusing update from remote peer 2edab657a33a4a0abfbd3262b1668483: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260925 03:03:06.683208 23749 consensus_queue.cc:1035] T c30e9a1092fe4d99994480aab8863cf6 P 2edab657a33a4a0abfbd3262b1668483 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260925 03:03:06.691535 23582 raft_consensus.cc:1261] T c30e9a1092fe4d99994480aab8863cf6 P b66c1d3fe3174f64b0936c512ee08f48 [term 4 FOLLOWER]: Refusing update from remote peer 2edab657a33a4a0abfbd3262b1668483: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260925 03:03:06.692927 23749 consensus_queue.cc:1035] T c30e9a1092fe4d99994480aab8863cf6 P 2edab657a33a4a0abfbd3262b1668483 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260925 03:03:06.801055 23582 raft_consensus.cc:1261] T 832e06098a3e499f82d53853e26235cf P b66c1d3fe3174f64b0936c512ee08f48 [term 3 FOLLOWER]: Refusing update from remote peer 2edab657a33a4a0abfbd3262b1668483: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260925 03:03:06.802204 23749 consensus_queue.cc:1035] T 832e06098a3e499f82d53853e26235cf P 2edab657a33a4a0abfbd3262b1668483 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260925 03:03:06.810612 23434 raft_consensus.cc:1261] T 832e06098a3e499f82d53853e26235cf P fa551b23283b4034b256ba25acec541b [term 3 FOLLOWER]: Refusing update from remote peer 2edab657a33a4a0abfbd3262b1668483: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260925 03:03:06.811965 23769 consensus_queue.cc:1035] T 832e06098a3e499f82d53853e26235cf P 2edab657a33a4a0abfbd3262b1668483 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260925 03:03:09.592139 23300 auto_leader_rebalancer-test.cc:303] The leader distribution is
I20260925 03:03:09.592897 23300 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260925 03:03:09.599319 23508 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "138dfa413a144a0e8911a2b402857cb4"
dest_uuid: "2edab657a33a4a0abfbd3262b1668483"
mode: GRACEFUL
new_leader_uuid: "fa551b23283b4034b256ba25acec541b"
from {username='slave'} at 127.0.0.1:45788
I20260925 03:03:09.599967 23508 raft_consensus.cc:595] T 138dfa413a144a0e8911a2b402857cb4 P 2edab657a33a4a0abfbd3262b1668483 [term 2 LEADER]: Received request to transfer leadership to TS fa551b23283b4034b256ba25acec541b
I20260925 03:03:09.602289 23508 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2baf62cd63564f4ba08b201e40e19d55"
dest_uuid: "2edab657a33a4a0abfbd3262b1668483"
mode: GRACEFUL
new_leader_uuid: "fa551b23283b4034b256ba25acec541b"
from {username='slave'} at 127.0.0.1:45788
I20260925 03:03:09.602728 23508 raft_consensus.cc:595] T 2baf62cd63564f4ba08b201e40e19d55 P 2edab657a33a4a0abfbd3262b1668483 [term 4 LEADER]: Received request to transfer leadership to TS fa551b23283b4034b256ba25acec541b
I20260925 03:03:09.604804 23508 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "832e06098a3e499f82d53853e26235cf"
dest_uuid: "2edab657a33a4a0abfbd3262b1668483"
mode: GRACEFUL
new_leader_uuid: "b66c1d3fe3174f64b0936c512ee08f48"
from {username='slave'} at 127.0.0.1:45788
I20260925 03:03:09.605197 23508 raft_consensus.cc:595] T 832e06098a3e499f82d53853e26235cf P 2edab657a33a4a0abfbd3262b1668483 [term 3 LEADER]: Received request to transfer leadership to TS b66c1d3fe3174f64b0936c512ee08f48
I20260925 03:03:09.607070 23508 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c30e9a1092fe4d99994480aab8863cf6"
dest_uuid: "2edab657a33a4a0abfbd3262b1668483"
mode: GRACEFUL
new_leader_uuid: "fa551b23283b4034b256ba25acec541b"
from {username='slave'} at 127.0.0.1:45788
I20260925 03:03:09.607462 23508 raft_consensus.cc:595] T c30e9a1092fe4d99994480aab8863cf6 P 2edab657a33a4a0abfbd3262b1668483 [term 4 LEADER]: Received request to transfer leadership to TS fa551b23283b4034b256ba25acec541b
I20260925 03:03:09.609253 23508 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d01220d66a8c4d34ac86fc3559fa5165"
dest_uuid: "2edab657a33a4a0abfbd3262b1668483"
mode: GRACEFUL
new_leader_uuid: "b66c1d3fe3174f64b0936c512ee08f48"
from {username='slave'} at 127.0.0.1:45788
I20260925 03:03:09.609625 23508 raft_consensus.cc:595] T d01220d66a8c4d34ac86fc3559fa5165 P 2edab657a33a4a0abfbd3262b1668483 [term 3 LEADER]: Received request to transfer leadership to TS b66c1d3fe3174f64b0936c512ee08f48
I20260925 03:03:09.610497 23300 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260925 03:03:09.610970 23300 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
W20260925 03:03:09.989017 23363 debug-util.cc:398] Leaking SignalData structure 0x7b0800060060 after lost signal to thread 23303
W20260925 03:03:09.989915 23363 debug-util.cc:398] Leaking SignalData structure 0x7b0800060740 after lost signal to thread 23366
W20260925 03:03:09.990648 23363 debug-util.cc:398] Leaking SignalData structure 0x7b08001fc820 after lost signal to thread 23459
W20260925 03:03:09.991317 23363 debug-util.cc:398] Leaking SignalData structure 0x7b08001dbea0 after lost signal to thread 23534
W20260925 03:03:09.991977 23363 debug-util.cc:398] Leaking SignalData structure 0x7b08001ca0a0 after lost signal to thread 23608
I20260925 03:03:10.041468 23781 raft_consensus.cc:979] T c30e9a1092fe4d99994480aab8863cf6 P 2edab657a33a4a0abfbd3262b1668483: : Instructing follower fa551b23283b4034b256ba25acec541b to start an election
I20260925 03:03:10.041994 23749 raft_consensus.cc:1067] T c30e9a1092fe4d99994480aab8863cf6 P 2edab657a33a4a0abfbd3262b1668483 [term 4 LEADER]: Signalling peer fa551b23283b4034b256ba25acec541b to start an election
I20260925 03:03:10.043705 23435 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c30e9a1092fe4d99994480aab8863cf6"
dest_uuid: "fa551b23283b4034b256ba25acec541b"
from {username='slave'} at 127.0.0.1:40538
I20260925 03:03:10.044263 23435 raft_consensus.cc:482] T c30e9a1092fe4d99994480aab8863cf6 P fa551b23283b4034b256ba25acec541b [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 03:03:10.044662 23435 raft_consensus.cc:3037] T c30e9a1092fe4d99994480aab8863cf6 P fa551b23283b4034b256ba25acec541b [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:03:10.052106 23435 raft_consensus.cc:504] T c30e9a1092fe4d99994480aab8863cf6 P fa551b23283b4034b256ba25acec541b [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:03:10.054946 23435 leader_election.cc:288] T c30e9a1092fe4d99994480aab8863cf6 P fa551b23283b4034b256ba25acec541b [CANDIDATE]: Term 5 election: Requested vote from peers 2edab657a33a4a0abfbd3262b1668483 (127.22.193.2:35431), b66c1d3fe3174f64b0936c512ee08f48 (127.22.193.3:42621)
I20260925 03:03:10.056324 23584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c30e9a1092fe4d99994480aab8863cf6" candidate_uuid: "fa551b23283b4034b256ba25acec541b" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "b66c1d3fe3174f64b0936c512ee08f48"
I20260925 03:03:10.056941 23749 raft_consensus.cc:979] T 2baf62cd63564f4ba08b201e40e19d55 P 2edab657a33a4a0abfbd3262b1668483: : Instructing follower fa551b23283b4034b256ba25acec541b to start an election
I20260925 03:03:10.057238 23584 raft_consensus.cc:3037] T c30e9a1092fe4d99994480aab8863cf6 P b66c1d3fe3174f64b0936c512ee08f48 [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:03:10.057418 23749 raft_consensus.cc:1067] T 2baf62cd63564f4ba08b201e40e19d55 P 2edab657a33a4a0abfbd3262b1668483 [term 4 LEADER]: Signalling peer fa551b23283b4034b256ba25acec541b to start an election
I20260925 03:03:10.057212 23508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c30e9a1092fe4d99994480aab8863cf6" candidate_uuid: "fa551b23283b4034b256ba25acec541b" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "2edab657a33a4a0abfbd3262b1668483"
I20260925 03:03:10.059078 23435 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2baf62cd63564f4ba08b201e40e19d55"
dest_uuid: "fa551b23283b4034b256ba25acec541b"
from {username='slave'} at 127.0.0.1:40538
I20260925 03:03:10.059669 23435 raft_consensus.cc:482] T 2baf62cd63564f4ba08b201e40e19d55 P fa551b23283b4034b256ba25acec541b [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 03:03:10.059177 23508 raft_consensus.cc:3032] T c30e9a1092fe4d99994480aab8863cf6 P 2edab657a33a4a0abfbd3262b1668483 [term 4 LEADER]: Stepping down as leader of term 4
I20260925 03:03:10.060041 23435 raft_consensus.cc:3037] T 2baf62cd63564f4ba08b201e40e19d55 P fa551b23283b4034b256ba25acec541b [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:03:10.060199 23508 raft_consensus.cc:729] T c30e9a1092fe4d99994480aab8863cf6 P 2edab657a33a4a0abfbd3262b1668483 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 2edab657a33a4a0abfbd3262b1668483, State: Running, Role: LEADER
I20260925 03:03:10.061275 23508 consensus_queue.cc:260] T c30e9a1092fe4d99994480aab8863cf6 P 2edab657a33a4a0abfbd3262b1668483 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:03:10.062769 23508 raft_consensus.cc:3037] T c30e9a1092fe4d99994480aab8863cf6 P 2edab657a33a4a0abfbd3262b1668483 [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:03:10.064946 23584 raft_consensus.cc:2454] T c30e9a1092fe4d99994480aab8863cf6 P b66c1d3fe3174f64b0936c512ee08f48 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa551b23283b4034b256ba25acec541b in term 5.
I20260925 03:03:10.066337 23398 leader_election.cc:302] T c30e9a1092fe4d99994480aab8863cf6 P fa551b23283b4034b256ba25acec541b [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b66c1d3fe3174f64b0936c512ee08f48, fa551b23283b4034b256ba25acec541b; no voters:
I20260925 03:03:10.066386 23435 raft_consensus.cc:504] T 2baf62cd63564f4ba08b201e40e19d55 P fa551b23283b4034b256ba25acec541b [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:03:10.068711 23435 leader_election.cc:288] T 2baf62cd63564f4ba08b201e40e19d55 P fa551b23283b4034b256ba25acec541b [CANDIDATE]: Term 5 election: Requested vote from peers 2edab657a33a4a0abfbd3262b1668483 (127.22.193.2:35431), b66c1d3fe3174f64b0936c512ee08f48 (127.22.193.3:42621)
I20260925 03:03:10.070830 23509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2baf62cd63564f4ba08b201e40e19d55" candidate_uuid: "fa551b23283b4034b256ba25acec541b" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "2edab657a33a4a0abfbd3262b1668483"
I20260925 03:03:10.071518 23509 raft_consensus.cc:3032] T 2baf62cd63564f4ba08b201e40e19d55 P 2edab657a33a4a0abfbd3262b1668483 [term 4 LEADER]: Stepping down as leader of term 4
I20260925 03:03:10.071841 23509 raft_consensus.cc:729] T 2baf62cd63564f4ba08b201e40e19d55 P 2edab657a33a4a0abfbd3262b1668483 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 2edab657a33a4a0abfbd3262b1668483, State: Running, Role: LEADER
I20260925 03:03:10.072613 23509 consensus_queue.cc:260] T 2baf62cd63564f4ba08b201e40e19d55 P 2edab657a33a4a0abfbd3262b1668483 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:03:10.073902 23509 raft_consensus.cc:3037] T 2baf62cd63564f4ba08b201e40e19d55 P 2edab657a33a4a0abfbd3262b1668483 [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:03:10.076393 23508 raft_consensus.cc:2454] T c30e9a1092fe4d99994480aab8863cf6 P 2edab657a33a4a0abfbd3262b1668483 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa551b23283b4034b256ba25acec541b in term 5.
I20260925 03:03:10.078441 23782 raft_consensus.cc:2781] T c30e9a1092fe4d99994480aab8863cf6 P fa551b23283b4034b256ba25acec541b [term 5 FOLLOWER]: Leader election won for term 5
I20260925 03:03:10.079450 23584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2baf62cd63564f4ba08b201e40e19d55" candidate_uuid: "fa551b23283b4034b256ba25acec541b" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "b66c1d3fe3174f64b0936c512ee08f48"
I20260925 03:03:10.080294 23584 raft_consensus.cc:3037] T 2baf62cd63564f4ba08b201e40e19d55 P b66c1d3fe3174f64b0936c512ee08f48 [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:03:10.082101 23509 raft_consensus.cc:2454] T 2baf62cd63564f4ba08b201e40e19d55 P 2edab657a33a4a0abfbd3262b1668483 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa551b23283b4034b256ba25acec541b in term 5.
I20260925 03:03:10.083408 23399 leader_election.cc:302] T 2baf62cd63564f4ba08b201e40e19d55 P fa551b23283b4034b256ba25acec541b [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2edab657a33a4a0abfbd3262b1668483, fa551b23283b4034b256ba25acec541b; no voters:
I20260925 03:03:10.084545 23782 raft_consensus.cc:686] T c30e9a1092fe4d99994480aab8863cf6 P fa551b23283b4034b256ba25acec541b [term 5 LEADER]: Becoming Leader. State: Replica: fa551b23283b4034b256ba25acec541b, State: Running, Role: LEADER
I20260925 03:03:10.085580 23782 consensus_queue.cc:237] T c30e9a1092fe4d99994480aab8863cf6 P fa551b23283b4034b256ba25acec541b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:03:10.086982 23781 raft_consensus.cc:979] T d01220d66a8c4d34ac86fc3559fa5165 P 2edab657a33a4a0abfbd3262b1668483: : Instructing follower b66c1d3fe3174f64b0936c512ee08f48 to start an election
I20260925 03:03:10.087412 23781 raft_consensus.cc:1067] T d01220d66a8c4d34ac86fc3559fa5165 P 2edab657a33a4a0abfbd3262b1668483 [term 3 LEADER]: Signalling peer b66c1d3fe3174f64b0936c512ee08f48 to start an election
I20260925 03:03:10.089316 23581 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d01220d66a8c4d34ac86fc3559fa5165"
dest_uuid: "b66c1d3fe3174f64b0936c512ee08f48"
from {username='slave'} at 127.0.0.1:51168
I20260925 03:03:10.089968 23581 raft_consensus.cc:482] T d01220d66a8c4d34ac86fc3559fa5165 P b66c1d3fe3174f64b0936c512ee08f48 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 03:03:10.090309 23581 raft_consensus.cc:3037] T d01220d66a8c4d34ac86fc3559fa5165 P b66c1d3fe3174f64b0936c512ee08f48 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:03:10.091127 23783 raft_consensus.cc:2781] T 2baf62cd63564f4ba08b201e40e19d55 P fa551b23283b4034b256ba25acec541b [term 5 FOLLOWER]: Leader election won for term 5
I20260925 03:03:10.091558 23783 raft_consensus.cc:686] T 2baf62cd63564f4ba08b201e40e19d55 P fa551b23283b4034b256ba25acec541b [term 5 LEADER]: Becoming Leader. State: Replica: fa551b23283b4034b256ba25acec541b, State: Running, Role: LEADER
I20260925 03:03:10.097663 23581 raft_consensus.cc:504] T d01220d66a8c4d34ac86fc3559fa5165 P b66c1d3fe3174f64b0936c512ee08f48 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:03:10.102214 23584 raft_consensus.cc:2454] T 2baf62cd63564f4ba08b201e40e19d55 P b66c1d3fe3174f64b0936c512ee08f48 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa551b23283b4034b256ba25acec541b in term 5.
I20260925 03:03:10.108034 23435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d01220d66a8c4d34ac86fc3559fa5165" candidate_uuid: "b66c1d3fe3174f64b0936c512ee08f48" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "fa551b23283b4034b256ba25acec541b"
I20260925 03:03:10.108454 23509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d01220d66a8c4d34ac86fc3559fa5165" candidate_uuid: "b66c1d3fe3174f64b0936c512ee08f48" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "2edab657a33a4a0abfbd3262b1668483"
I20260925 03:03:10.108620 23435 raft_consensus.cc:3037] T d01220d66a8c4d34ac86fc3559fa5165 P fa551b23283b4034b256ba25acec541b [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:03:10.109025 23509 raft_consensus.cc:3032] T d01220d66a8c4d34ac86fc3559fa5165 P 2edab657a33a4a0abfbd3262b1668483 [term 3 LEADER]: Stepping down as leader of term 3
I20260925 03:03:10.109792 23581 leader_election.cc:288] T d01220d66a8c4d34ac86fc3559fa5165 P b66c1d3fe3174f64b0936c512ee08f48 [CANDIDATE]: Term 4 election: Requested vote from peers fa551b23283b4034b256ba25acec541b (127.22.193.1:42181), 2edab657a33a4a0abfbd3262b1668483 (127.22.193.2:35431)
I20260925 03:03:10.109205 23331 catalog_manager.cc:5495] T c30e9a1092fe4d99994480aab8863cf6 P fa551b23283b4034b256ba25acec541b reported cstate change: term changed from 4 to 5, leader changed from 2edab657a33a4a0abfbd3262b1668483 (127.22.193.2) to fa551b23283b4034b256ba25acec541b (127.22.193.1). New cstate: current_term: 5 leader_uuid: "fa551b23283b4034b256ba25acec541b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } health_report { overall_health: UNKNOWN } } }
I20260925 03:03:10.109351 23509 raft_consensus.cc:729] T d01220d66a8c4d34ac86fc3559fa5165 P 2edab657a33a4a0abfbd3262b1668483 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 2edab657a33a4a0abfbd3262b1668483, State: Running, Role: LEADER
I20260925 03:03:10.115506 23779 raft_consensus.cc:979] T 832e06098a3e499f82d53853e26235cf P 2edab657a33a4a0abfbd3262b1668483: : Instructing follower b66c1d3fe3174f64b0936c512ee08f48 to start an election
I20260925 03:03:10.116288 23749 raft_consensus.cc:1067] T 832e06098a3e499f82d53853e26235cf P 2edab657a33a4a0abfbd3262b1668483 [term 3 LEADER]: Signalling peer b66c1d3fe3174f64b0936c512ee08f48 to start an election
I20260925 03:03:10.117031 23435 raft_consensus.cc:2454] T d01220d66a8c4d34ac86fc3559fa5165 P fa551b23283b4034b256ba25acec541b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b66c1d3fe3174f64b0936c512ee08f48 in term 4.
I20260925 03:03:10.117944 23509 consensus_queue.cc:260] T d01220d66a8c4d34ac86fc3559fa5165 P 2edab657a33a4a0abfbd3262b1668483 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:03:10.119346 23509 raft_consensus.cc:3037] T d01220d66a8c4d34ac86fc3559fa5165 P 2edab657a33a4a0abfbd3262b1668483 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:03:10.122958 23547 leader_election.cc:302] T d01220d66a8c4d34ac86fc3559fa5165 P b66c1d3fe3174f64b0936c512ee08f48 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b66c1d3fe3174f64b0936c512ee08f48, fa551b23283b4034b256ba25acec541b; no voters:
I20260925 03:03:10.123997 23780 raft_consensus.cc:2781] T d01220d66a8c4d34ac86fc3559fa5165 P b66c1d3fe3174f64b0936c512ee08f48 [term 4 FOLLOWER]: Leader election won for term 4
I20260925 03:03:10.126744 23509 raft_consensus.cc:2454] T d01220d66a8c4d34ac86fc3559fa5165 P 2edab657a33a4a0abfbd3262b1668483 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b66c1d3fe3174f64b0936c512ee08f48 in term 4.
I20260925 03:03:10.127377 23781 raft_consensus.cc:979] T 138dfa413a144a0e8911a2b402857cb4 P 2edab657a33a4a0abfbd3262b1668483: : Instructing follower fa551b23283b4034b256ba25acec541b to start an election
I20260925 03:03:10.127844 23781 raft_consensus.cc:1067] T 138dfa413a144a0e8911a2b402857cb4 P 2edab657a33a4a0abfbd3262b1668483 [term 2 LEADER]: Signalling peer fa551b23283b4034b256ba25acec541b to start an election
I20260925 03:03:10.130064 23432 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "138dfa413a144a0e8911a2b402857cb4"
dest_uuid: "fa551b23283b4034b256ba25acec541b"
from {username='slave'} at 127.0.0.1:40538
I20260925 03:03:10.130723 23432 raft_consensus.cc:482] T 138dfa413a144a0e8911a2b402857cb4 P fa551b23283b4034b256ba25acec541b [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 03:03:10.131117 23432 raft_consensus.cc:3037] T 138dfa413a144a0e8911a2b402857cb4 P fa551b23283b4034b256ba25acec541b [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:03:10.122113 23584 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "832e06098a3e499f82d53853e26235cf"
dest_uuid: "b66c1d3fe3174f64b0936c512ee08f48"
from {username='slave'} at 127.0.0.1:51168
I20260925 03:03:10.133332 23584 raft_consensus.cc:482] T 832e06098a3e499f82d53853e26235cf P b66c1d3fe3174f64b0936c512ee08f48 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 03:03:10.133728 23584 raft_consensus.cc:3037] T 832e06098a3e499f82d53853e26235cf P b66c1d3fe3174f64b0936c512ee08f48 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:03:10.137452 23780 raft_consensus.cc:686] T d01220d66a8c4d34ac86fc3559fa5165 P b66c1d3fe3174f64b0936c512ee08f48 [term 4 LEADER]: Becoming Leader. State: Replica: b66c1d3fe3174f64b0936c512ee08f48, State: Running, Role: LEADER
I20260925 03:03:10.138542 23780 consensus_queue.cc:237] T d01220d66a8c4d34ac86fc3559fa5165 P b66c1d3fe3174f64b0936c512ee08f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:03:10.142908 23432 raft_consensus.cc:504] T 138dfa413a144a0e8911a2b402857cb4 P fa551b23283b4034b256ba25acec541b [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:03:10.149574 23783 consensus_queue.cc:237] T 2baf62cd63564f4ba08b201e40e19d55 P fa551b23283b4034b256ba25acec541b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:03:10.152920 23509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "138dfa413a144a0e8911a2b402857cb4" candidate_uuid: "fa551b23283b4034b256ba25acec541b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "2edab657a33a4a0abfbd3262b1668483"
I20260925 03:03:10.153510 23509 raft_consensus.cc:3032] T 138dfa413a144a0e8911a2b402857cb4 P 2edab657a33a4a0abfbd3262b1668483 [term 2 LEADER]: Stepping down as leader of term 2
I20260925 03:03:10.153818 23509 raft_consensus.cc:729] T 138dfa413a144a0e8911a2b402857cb4 P 2edab657a33a4a0abfbd3262b1668483 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2edab657a33a4a0abfbd3262b1668483, State: Running, Role: LEADER
I20260925 03:03:10.154559 23509 consensus_queue.cc:260] T 138dfa413a144a0e8911a2b402857cb4 P 2edab657a33a4a0abfbd3262b1668483 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:03:10.148106 23584 raft_consensus.cc:504] T 832e06098a3e499f82d53853e26235cf P b66c1d3fe3174f64b0936c512ee08f48 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:03:10.155714 23509 raft_consensus.cc:3037] T 138dfa413a144a0e8911a2b402857cb4 P 2edab657a33a4a0abfbd3262b1668483 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:03:10.157986 23432 leader_election.cc:288] T 138dfa413a144a0e8911a2b402857cb4 P fa551b23283b4034b256ba25acec541b [CANDIDATE]: Term 3 election: Requested vote from peers 2edab657a33a4a0abfbd3262b1668483 (127.22.193.2:35431), b66c1d3fe3174f64b0936c512ee08f48 (127.22.193.3:42621)
I20260925 03:03:10.158232 23581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "138dfa413a144a0e8911a2b402857cb4" candidate_uuid: "fa551b23283b4034b256ba25acec541b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "b66c1d3fe3174f64b0936c512ee08f48"
I20260925 03:03:10.158805 23581 raft_consensus.cc:3037] T 138dfa413a144a0e8911a2b402857cb4 P b66c1d3fe3174f64b0936c512ee08f48 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:03:10.160418 23432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "832e06098a3e499f82d53853e26235cf" candidate_uuid: "b66c1d3fe3174f64b0936c512ee08f48" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "fa551b23283b4034b256ba25acec541b"
I20260925 03:03:10.161036 23432 raft_consensus.cc:3037] T 832e06098a3e499f82d53853e26235cf P fa551b23283b4034b256ba25acec541b [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:03:10.163154 23332 catalog_manager.cc:5495] T d01220d66a8c4d34ac86fc3559fa5165 P b66c1d3fe3174f64b0936c512ee08f48 reported cstate change: term changed from 3 to 4, leader changed from 2edab657a33a4a0abfbd3262b1668483 (127.22.193.2) to b66c1d3fe3174f64b0936c512ee08f48 (127.22.193.3). New cstate: current_term: 4 leader_uuid: "b66c1d3fe3174f64b0936c512ee08f48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } health_report { overall_health: HEALTHY } } }
I20260925 03:03:10.165656 23581 raft_consensus.cc:2454] T 138dfa413a144a0e8911a2b402857cb4 P b66c1d3fe3174f64b0936c512ee08f48 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa551b23283b4034b256ba25acec541b in term 3.
I20260925 03:03:10.165537 23584 leader_election.cc:288] T 832e06098a3e499f82d53853e26235cf P b66c1d3fe3174f64b0936c512ee08f48 [CANDIDATE]: Term 4 election: Requested vote from peers fa551b23283b4034b256ba25acec541b (127.22.193.1:42181), 2edab657a33a4a0abfbd3262b1668483 (127.22.193.2:35431)
I20260925 03:03:10.166841 23398 leader_election.cc:302] T 138dfa413a144a0e8911a2b402857cb4 P fa551b23283b4034b256ba25acec541b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b66c1d3fe3174f64b0936c512ee08f48, fa551b23283b4034b256ba25acec541b; no voters:
I20260925 03:03:10.168082 23784 raft_consensus.cc:2781] T 138dfa413a144a0e8911a2b402857cb4 P fa551b23283b4034b256ba25acec541b [term 3 FOLLOWER]: Leader election won for term 3
I20260925 03:03:10.168568 23784 raft_consensus.cc:686] T 138dfa413a144a0e8911a2b402857cb4 P fa551b23283b4034b256ba25acec541b [term 3 LEADER]: Becoming Leader. State: Replica: fa551b23283b4034b256ba25acec541b, State: Running, Role: LEADER
I20260925 03:03:10.168536 23508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "832e06098a3e499f82d53853e26235cf" candidate_uuid: "b66c1d3fe3174f64b0936c512ee08f48" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "2edab657a33a4a0abfbd3262b1668483"
I20260925 03:03:10.169723 23508 raft_consensus.cc:3032] T 832e06098a3e499f82d53853e26235cf P 2edab657a33a4a0abfbd3262b1668483 [term 3 LEADER]: Stepping down as leader of term 3
I20260925 03:03:10.170064 23508 raft_consensus.cc:729] T 832e06098a3e499f82d53853e26235cf P 2edab657a33a4a0abfbd3262b1668483 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 2edab657a33a4a0abfbd3262b1668483, State: Running, Role: LEADER
I20260925 03:03:10.169637 23784 consensus_queue.cc:237] T 138dfa413a144a0e8911a2b402857cb4 P fa551b23283b4034b256ba25acec541b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:03:10.170735 23508 consensus_queue.cc:260] T 832e06098a3e499f82d53853e26235cf P 2edab657a33a4a0abfbd3262b1668483 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:03:10.171973 23508 raft_consensus.cc:3037] T 832e06098a3e499f82d53853e26235cf P 2edab657a33a4a0abfbd3262b1668483 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:03:10.181905 23508 raft_consensus.cc:2454] T 832e06098a3e499f82d53853e26235cf P 2edab657a33a4a0abfbd3262b1668483 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b66c1d3fe3174f64b0936c512ee08f48 in term 4.
I20260925 03:03:10.183162 23548 leader_election.cc:302] T 832e06098a3e499f82d53853e26235cf P b66c1d3fe3174f64b0936c512ee08f48 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2edab657a33a4a0abfbd3262b1668483, b66c1d3fe3174f64b0936c512ee08f48; no voters:
I20260925 03:03:10.183498 23432 raft_consensus.cc:2454] T 832e06098a3e499f82d53853e26235cf P fa551b23283b4034b256ba25acec541b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b66c1d3fe3174f64b0936c512ee08f48 in term 4.
I20260925 03:03:10.183966 23780 raft_consensus.cc:2781] T 832e06098a3e499f82d53853e26235cf P b66c1d3fe3174f64b0936c512ee08f48 [term 4 FOLLOWER]: Leader election won for term 4
I20260925 03:03:10.184437 23780 raft_consensus.cc:686] T 832e06098a3e499f82d53853e26235cf P b66c1d3fe3174f64b0936c512ee08f48 [term 4 LEADER]: Becoming Leader. State: Replica: b66c1d3fe3174f64b0936c512ee08f48, State: Running, Role: LEADER
I20260925 03:03:10.184451 23509 raft_consensus.cc:2454] T 138dfa413a144a0e8911a2b402857cb4 P 2edab657a33a4a0abfbd3262b1668483 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa551b23283b4034b256ba25acec541b in term 3.
I20260925 03:03:10.185613 23780 consensus_queue.cc:237] T 832e06098a3e499f82d53853e26235cf P b66c1d3fe3174f64b0936c512ee08f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } }
I20260925 03:03:10.205281 23331 catalog_manager.cc:5495] T 2baf62cd63564f4ba08b201e40e19d55 P fa551b23283b4034b256ba25acec541b reported cstate change: term changed from 4 to 5, leader changed from 2edab657a33a4a0abfbd3262b1668483 (127.22.193.2) to fa551b23283b4034b256ba25acec541b (127.22.193.1). New cstate: current_term: 5 leader_uuid: "fa551b23283b4034b256ba25acec541b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } health_report { overall_health: UNKNOWN } } }
I20260925 03:03:10.205144 23332 catalog_manager.cc:5495] T 832e06098a3e499f82d53853e26235cf P b66c1d3fe3174f64b0936c512ee08f48 reported cstate change: term changed from 3 to 4, leader changed from 2edab657a33a4a0abfbd3262b1668483 (127.22.193.2) to b66c1d3fe3174f64b0936c512ee08f48 (127.22.193.3). New cstate: current_term: 4 leader_uuid: "b66c1d3fe3174f64b0936c512ee08f48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } health_report { overall_health: HEALTHY } } }
I20260925 03:03:10.206488 23331 catalog_manager.cc:5495] T 138dfa413a144a0e8911a2b402857cb4 P fa551b23283b4034b256ba25acec541b reported cstate change: term changed from 2 to 3, leader changed from 2edab657a33a4a0abfbd3262b1668483 (127.22.193.2) to fa551b23283b4034b256ba25acec541b (127.22.193.1). New cstate: current_term: 3 leader_uuid: "fa551b23283b4034b256ba25acec541b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 } health_report { overall_health: UNKNOWN } } }
I20260925 03:03:10.493520 23584 raft_consensus.cc:1261] T c30e9a1092fe4d99994480aab8863cf6 P b66c1d3fe3174f64b0936c512ee08f48 [term 5 FOLLOWER]: Refusing update from remote peer fa551b23283b4034b256ba25acec541b: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260925 03:03:10.494979 23784 consensus_queue.cc:1035] T c30e9a1092fe4d99994480aab8863cf6 P fa551b23283b4034b256ba25acec541b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260925 03:03:10.511889 23509 raft_consensus.cc:1261] T c30e9a1092fe4d99994480aab8863cf6 P 2edab657a33a4a0abfbd3262b1668483 [term 5 FOLLOWER]: Refusing update from remote peer fa551b23283b4034b256ba25acec541b: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260925 03:03:10.514905 23783 consensus_queue.cc:1035] T c30e9a1092fe4d99994480aab8863cf6 P fa551b23283b4034b256ba25acec541b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260925 03:03:10.556290 23584 raft_consensus.cc:1261] T 138dfa413a144a0e8911a2b402857cb4 P b66c1d3fe3174f64b0936c512ee08f48 [term 3 FOLLOWER]: Refusing update from remote peer fa551b23283b4034b256ba25acec541b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260925 03:03:10.557521 23783 consensus_queue.cc:1035] T 138dfa413a144a0e8911a2b402857cb4 P fa551b23283b4034b256ba25acec541b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260925 03:03:10.568365 23509 raft_consensus.cc:1261] T 138dfa413a144a0e8911a2b402857cb4 P 2edab657a33a4a0abfbd3262b1668483 [term 3 FOLLOWER]: Refusing update from remote peer fa551b23283b4034b256ba25acec541b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260925 03:03:10.570641 23784 consensus_queue.cc:1035] T 138dfa413a144a0e8911a2b402857cb4 P fa551b23283b4034b256ba25acec541b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260925 03:03:10.588064 23508 raft_consensus.cc:1261] T 832e06098a3e499f82d53853e26235cf P 2edab657a33a4a0abfbd3262b1668483 [term 4 FOLLOWER]: Refusing update from remote peer b66c1d3fe3174f64b0936c512ee08f48: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260925 03:03:10.589298 23780 consensus_queue.cc:1035] T 832e06098a3e499f82d53853e26235cf P b66c1d3fe3174f64b0936c512ee08f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260925 03:03:10.601218 23432 raft_consensus.cc:1261] T 832e06098a3e499f82d53853e26235cf P fa551b23283b4034b256ba25acec541b [term 4 FOLLOWER]: Refusing update from remote peer b66c1d3fe3174f64b0936c512ee08f48: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260925 03:03:10.602968 23780 consensus_queue.cc:1035] T 832e06098a3e499f82d53853e26235cf P b66c1d3fe3174f64b0936c512ee08f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260925 03:03:10.612553 23300 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260925 03:03:10.617111 23508 raft_consensus.cc:1261] T 2baf62cd63564f4ba08b201e40e19d55 P 2edab657a33a4a0abfbd3262b1668483 [term 5 FOLLOWER]: Refusing update from remote peer fa551b23283b4034b256ba25acec541b: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260925 03:03:10.618461 23783 consensus_queue.cc:1035] T 2baf62cd63564f4ba08b201e40e19d55 P fa551b23283b4034b256ba25acec541b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260925 03:03:10.624109 23300 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260925 03:03:10.624677 23300 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260925 03:03:10.631927 23584 raft_consensus.cc:1261] T 2baf62cd63564f4ba08b201e40e19d55 P b66c1d3fe3174f64b0936c512ee08f48 [term 5 FOLLOWER]: Refusing update from remote peer fa551b23283b4034b256ba25acec541b: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260925 03:03:10.634865 23782 consensus_queue.cc:1035] T 2baf62cd63564f4ba08b201e40e19d55 P fa551b23283b4034b256ba25acec541b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b66c1d3fe3174f64b0936c512ee08f48" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 42621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260925 03:03:10.647338 23508 raft_consensus.cc:1261] T d01220d66a8c4d34ac86fc3559fa5165 P 2edab657a33a4a0abfbd3262b1668483 [term 4 FOLLOWER]: Refusing update from remote peer b66c1d3fe3174f64b0936c512ee08f48: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260925 03:03:10.648448 23780 consensus_queue.cc:1035] T d01220d66a8c4d34ac86fc3559fa5165 P b66c1d3fe3174f64b0936c512ee08f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2edab657a33a4a0abfbd3262b1668483" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 35431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260925 03:03:10.659652 23432 raft_consensus.cc:1261] T d01220d66a8c4d34ac86fc3559fa5165 P fa551b23283b4034b256ba25acec541b [term 4 FOLLOWER]: Refusing update from remote peer b66c1d3fe3174f64b0936c512ee08f48: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260925 03:03:10.663538 23786 consensus_queue.cc:1035] T d01220d66a8c4d34ac86fc3559fa5165 P b66c1d3fe3174f64b0936c512ee08f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa551b23283b4034b256ba25acec541b" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260925 03:03:11.625706 23300 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260925 03:03:11.633102 23300 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260925 03:03:11.633608 23300 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260925 03:03:12.634531 23300 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260925 03:03:12.639611 23300 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260925 03:03:12.640041 23300 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260925 03:03:13.641136 23300 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260925 03:03:13.646003 23300 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260925 03:03:13.646466 23300 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260925 03:03:14.647230 23300 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260925 03:03:14.651980 23300 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260925 03:03:14.652400 23300 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260925 03:03:15.653230 23300 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260925 03:03:15.658280 23300 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260925 03:03:15.658730 23300 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260925 03:03:16.659634 23300 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260925 03:03:16.665215 23300 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260925 03:03:16.665810 23300 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260925 03:03:17.666988 23300 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260925 03:03:17.673800 23300 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260925 03:03:17.674261 23300 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260925 03:03:18.675572 23300 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260925 03:03:18.681362 23300 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260925 03:03:18.681836 23300 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260925 03:03:19.710952 23300 auto_leader_rebalancer-test.cc:314] The leader distribution is
I20260925 03:03:19.711515 23300 tablet_server.cc:178] TabletServer@127.22.193.1:0 shutting down...
I20260925 03:03:19.737665 23300 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260925 03:03:19.738353 23300 tablet_replica.cc:330] T c30e9a1092fe4d99994480aab8863cf6 P fa551b23283b4034b256ba25acec541b: stopping tablet replica
I20260925 03:03:19.739136 23300 raft_consensus.cc:2229] T c30e9a1092fe4d99994480aab8863cf6 P fa551b23283b4034b256ba25acec541b [term 5 LEADER]: Raft consensus shutting down.
I20260925 03:03:19.740113 23300 raft_consensus.cc:2258] T c30e9a1092fe4d99994480aab8863cf6 P fa551b23283b4034b256ba25acec541b [term 5 FOLLOWER]: Raft consensus is shut down!
I20260925 03:03:19.742949 23300 tablet_replica.cc:330] T 138dfa413a144a0e8911a2b402857cb4 P fa551b23283b4034b256ba25acec541b: stopping tablet replica
I20260925 03:03:19.743438 23300 raft_consensus.cc:2229] T 138dfa413a144a0e8911a2b402857cb4 P fa551b23283b4034b256ba25acec541b [term 3 LEADER]: Raft consensus shutting down.
I20260925 03:03:19.744174 23300 raft_consensus.cc:2258] T 138dfa413a144a0e8911a2b402857cb4 P fa551b23283b4034b256ba25acec541b [term 3 FOLLOWER]: Raft consensus is shut down!
I20260925 03:03:19.745815 23300 tablet_replica.cc:330] T 2baf62cd63564f4ba08b201e40e19d55 P fa551b23283b4034b256ba25acec541b: stopping tablet replica
I20260925 03:03:19.746285 23300 raft_consensus.cc:2229] T 2baf62cd63564f4ba08b201e40e19d55 P fa551b23283b4034b256ba25acec541b [term 5 LEADER]: Raft consensus shutting down.
I20260925 03:03:19.747082 23300 raft_consensus.cc:2258] T 2baf62cd63564f4ba08b201e40e19d55 P fa551b23283b4034b256ba25acec541b [term 5 FOLLOWER]: Raft consensus is shut down!
I20260925 03:03:19.748910 23300 tablet_replica.cc:330] T 832e06098a3e499f82d53853e26235cf P fa551b23283b4034b256ba25acec541b: stopping tablet replica
I20260925 03:03:19.749351 23300 raft_consensus.cc:2229] T 832e06098a3e499f82d53853e26235cf P fa551b23283b4034b256ba25acec541b [term 4 FOLLOWER]: Raft consensus shutting down.
I20260925 03:03:19.749733 23300 raft_consensus.cc:2258] T 832e06098a3e499f82d53853e26235cf P fa551b23283b4034b256ba25acec541b [term 4 FOLLOWER]: Raft consensus is shut down!
I20260925 03:03:19.751426 23300 tablet_replica.cc:330] T 9de619025d19432c8df31a19bb6fc446 P fa551b23283b4034b256ba25acec541b: stopping tablet replica
I20260925 03:03:19.751901 23300 raft_consensus.cc:2229] T 9de619025d19432c8df31a19bb6fc446 P fa551b23283b4034b256ba25acec541b [term 2 FOLLOWER]: Raft consensus shutting down.
I20260925 03:03:19.752297 23300 raft_consensus.cc:2258] T 9de619025d19432c8df31a19bb6fc446 P fa551b23283b4034b256ba25acec541b [term 2 FOLLOWER]: Raft consensus is shut down!
I20260925 03:03:19.753921 23300 tablet_replica.cc:330] T 43de311d912b475eb01e1f7c9a65412a P fa551b23283b4034b256ba25acec541b: stopping tablet replica
I20260925 03:03:19.754354 23300 raft_consensus.cc:2229] T 43de311d912b475eb01e1f7c9a65412a P fa551b23283b4034b256ba25acec541b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 03:03:19.754702 23300 raft_consensus.cc:2258] T 43de311d912b475eb01e1f7c9a65412a P fa551b23283b4034b256ba25acec541b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 03:03:19.756215 23300 tablet_replica.cc:330] T fb663a8b564c494f98f56845f8a05d6c P fa551b23283b4034b256ba25acec541b: stopping tablet replica
I20260925 03:03:19.756654 23300 raft_consensus.cc:2229] T fb663a8b564c494f98f56845f8a05d6c P fa551b23283b4034b256ba25acec541b [term 2 FOLLOWER]: Raft consensus shutting down.
I20260925 03:03:19.757153 23300 raft_consensus.cc:2258] T fb663a8b564c494f98f56845f8a05d6c P fa551b23283b4034b256ba25acec541b [term 2 FOLLOWER]: Raft consensus is shut down!
I20260925 03:03:19.758705 23300 tablet_replica.cc:330] T d01220d66a8c4d34ac86fc3559fa5165 P fa551b23283b4034b256ba25acec541b: stopping tablet replica
I20260925 03:03:19.759138 23300 raft_consensus.cc:2229] T d01220d66a8c4d34ac86fc3559fa5165 P fa551b23283b4034b256ba25acec541b [term 4 FOLLOWER]: Raft consensus shutting down.
I20260925 03:03:19.759546 23300 raft_consensus.cc:2258] T d01220d66a8c4d34ac86fc3559fa5165 P fa551b23283b4034b256ba25acec541b [term 4 FOLLOWER]: Raft consensus is shut down!
I20260925 03:03:19.761356 23300 tablet_replica.cc:330] T ef74cafe53e34e1dace8b02d73693521 P fa551b23283b4034b256ba25acec541b: stopping tablet replica
I20260925 03:03:19.761826 23300 raft_consensus.cc:2229] T ef74cafe53e34e1dace8b02d73693521 P fa551b23283b4034b256ba25acec541b [term 3 FOLLOWER]: Raft consensus shutting down.
I20260925 03:03:19.762243 23300 raft_consensus.cc:2258] T ef74cafe53e34e1dace8b02d73693521 P fa551b23283b4034b256ba25acec541b [term 3 FOLLOWER]: Raft consensus is shut down!
W20260925 03:03:19.866480 23547 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.193.1:42181: connect: Connection refused (error 111)
W20260925 03:03:19.871882 23547 consensus_peers.cc:487] T d01220d66a8c4d34ac86fc3559fa5165 P b66c1d3fe3174f64b0936c512ee08f48 -> Peer fa551b23283b4034b256ba25acec541b (127.22.193.1:42181): Couldn't send request to peer fa551b23283b4034b256ba25acec541b. Status: Network error: Client connection negotiation failed: client connection to 127.22.193.1:42181: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 03:03:19.874249 23300 tablet_server.cc:194] TabletServer@127.22.193.1:0 shutdown complete.
W20260925 03:03:19.874593 23473 consensus_peers.cc:487] T 43de311d912b475eb01e1f7c9a65412a P 2edab657a33a4a0abfbd3262b1668483 -> Peer fa551b23283b4034b256ba25acec541b (127.22.193.1:42181): Couldn't send request to peer fa551b23283b4034b256ba25acec541b. Status: Network error: Client connection negotiation failed: client connection to 127.22.193.1:42181: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260925 03:03:19.875161 23547 consensus_peers.cc:487] T 9de619025d19432c8df31a19bb6fc446 P b66c1d3fe3174f64b0936c512ee08f48 -> Peer fa551b23283b4034b256ba25acec541b (127.22.193.1:42181): Couldn't send request to peer fa551b23283b4034b256ba25acec541b. Status: Network error: Client connection negotiation failed: client connection to 127.22.193.1:42181: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 03:03:19.893307 23300 tablet_server.cc:178] TabletServer@127.22.193.2:0 shutting down...
W20260925 03:03:19.912425 23473 consensus_peers.cc:487] T fb663a8b564c494f98f56845f8a05d6c P 2edab657a33a4a0abfbd3262b1668483 -> Peer fa551b23283b4034b256ba25acec541b (127.22.193.1:42181): Couldn't send request to peer fa551b23283b4034b256ba25acec541b. Status: Network error: Client connection negotiation failed: client connection to 127.22.193.1:42181: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 03:03:19.919217 23300 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260925 03:03:19.920027 23300 tablet_replica.cc:330] T 832e06098a3e499f82d53853e26235cf P 2edab657a33a4a0abfbd3262b1668483: stopping tablet replica
I20260925 03:03:19.920783 23300 raft_consensus.cc:2229] T 832e06098a3e499f82d53853e26235cf P 2edab657a33a4a0abfbd3262b1668483 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260925 03:03:19.921465 23300 raft_consensus.cc:2258] T 832e06098a3e499f82d53853e26235cf P 2edab657a33a4a0abfbd3262b1668483 [term 4 FOLLOWER]: Raft consensus is shut down!
W20260925 03:03:19.921559 23548 consensus_peers.cc:487] T 832e06098a3e499f82d53853e26235cf P b66c1d3fe3174f64b0936c512ee08f48 -> Peer 2edab657a33a4a0abfbd3262b1668483 (127.22.193.2:35431): Couldn't send request to peer 2edab657a33a4a0abfbd3262b1668483. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260925 03:03:19.923578 23300 tablet_replica.cc:330] T 138dfa413a144a0e8911a2b402857cb4 P 2edab657a33a4a0abfbd3262b1668483: stopping tablet replica
I20260925 03:03:19.924091 23300 raft_consensus.cc:2229] T 138dfa413a144a0e8911a2b402857cb4 P 2edab657a33a4a0abfbd3262b1668483 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260925 03:03:19.924484 23300 raft_consensus.cc:2258] T 138dfa413a144a0e8911a2b402857cb4 P 2edab657a33a4a0abfbd3262b1668483 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260925 03:03:19.926234 23300 tablet_replica.cc:330] T c30e9a1092fe4d99994480aab8863cf6 P 2edab657a33a4a0abfbd3262b1668483: stopping tablet replica
I20260925 03:03:19.926692 23300 raft_consensus.cc:2229] T c30e9a1092fe4d99994480aab8863cf6 P 2edab657a33a4a0abfbd3262b1668483 [term 5 FOLLOWER]: Raft consensus shutting down.
I20260925 03:03:19.927057 23300 raft_consensus.cc:2258] T c30e9a1092fe4d99994480aab8863cf6 P 2edab657a33a4a0abfbd3262b1668483 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260925 03:03:19.928583 23300 tablet_replica.cc:330] T 43de311d912b475eb01e1f7c9a65412a P 2edab657a33a4a0abfbd3262b1668483: stopping tablet replica
I20260925 03:03:19.929081 23300 raft_consensus.cc:2229] T 43de311d912b475eb01e1f7c9a65412a P 2edab657a33a4a0abfbd3262b1668483 [term 1 LEADER]: Raft consensus shutting down.
I20260925 03:03:19.929805 23300 raft_consensus.cc:2258] T 43de311d912b475eb01e1f7c9a65412a P 2edab657a33a4a0abfbd3262b1668483 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 03:03:19.931456 23300 tablet_replica.cc:330] T 9de619025d19432c8df31a19bb6fc446 P 2edab657a33a4a0abfbd3262b1668483: stopping tablet replica
I20260925 03:03:19.931924 23300 raft_consensus.cc:2229] T 9de619025d19432c8df31a19bb6fc446 P 2edab657a33a4a0abfbd3262b1668483 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260925 03:03:19.932300 23300 raft_consensus.cc:2258] T 9de619025d19432c8df31a19bb6fc446 P 2edab657a33a4a0abfbd3262b1668483 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260925 03:03:19.933849 23300 tablet_replica.cc:330] T 2baf62cd63564f4ba08b201e40e19d55 P 2edab657a33a4a0abfbd3262b1668483: stopping tablet replica
I20260925 03:03:19.934340 23300 raft_consensus.cc:2229] T 2baf62cd63564f4ba08b201e40e19d55 P 2edab657a33a4a0abfbd3262b1668483 [term 5 FOLLOWER]: Raft consensus shutting down.
I20260925 03:03:19.934733 23300 raft_consensus.cc:2258] T 2baf62cd63564f4ba08b201e40e19d55 P 2edab657a33a4a0abfbd3262b1668483 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260925 03:03:19.936204 23300 tablet_replica.cc:330] T fb663a8b564c494f98f56845f8a05d6c P 2edab657a33a4a0abfbd3262b1668483: stopping tablet replica
I20260925 03:03:19.936650 23300 raft_consensus.cc:2229] T fb663a8b564c494f98f56845f8a05d6c P 2edab657a33a4a0abfbd3262b1668483 [term 2 LEADER]: Raft consensus shutting down.
I20260925 03:03:19.937512 23300 raft_consensus.cc:2258] T fb663a8b564c494f98f56845f8a05d6c P 2edab657a33a4a0abfbd3262b1668483 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260925 03:03:19.939015 23300 tablet_replica.cc:330] T d01220d66a8c4d34ac86fc3559fa5165 P 2edab657a33a4a0abfbd3262b1668483: stopping tablet replica
I20260925 03:03:19.939463 23300 raft_consensus.cc:2229] T d01220d66a8c4d34ac86fc3559fa5165 P 2edab657a33a4a0abfbd3262b1668483 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260925 03:03:19.939849 23300 raft_consensus.cc:2258] T d01220d66a8c4d34ac86fc3559fa5165 P 2edab657a33a4a0abfbd3262b1668483 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260925 03:03:19.941565 23300 tablet_replica.cc:330] T ef74cafe53e34e1dace8b02d73693521 P 2edab657a33a4a0abfbd3262b1668483: stopping tablet replica
I20260925 03:03:19.941980 23300 raft_consensus.cc:2229] T ef74cafe53e34e1dace8b02d73693521 P 2edab657a33a4a0abfbd3262b1668483 [term 3 LEADER]: Raft consensus shutting down.
I20260925 03:03:19.942766 23300 raft_consensus.cc:2258] T ef74cafe53e34e1dace8b02d73693521 P 2edab657a33a4a0abfbd3262b1668483 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260925 03:03:19.970162 23300 tablet_server.cc:194] TabletServer@127.22.193.2:0 shutdown complete.
I20260925 03:03:19.986943 23300 tablet_server.cc:178] TabletServer@127.22.193.3:0 shutting down...
W20260925 03:03:19.999580 23548 consensus_peers.cc:487] T d01220d66a8c4d34ac86fc3559fa5165 P b66c1d3fe3174f64b0936c512ee08f48 -> Peer 2edab657a33a4a0abfbd3262b1668483 (127.22.193.2:35431): Couldn't send request to peer 2edab657a33a4a0abfbd3262b1668483. Status: Network error: Client connection negotiation failed: client connection to 127.22.193.2:35431: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 03:03:20.012816 23300 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260925 03:03:20.013597 23300 tablet_replica.cc:330] T 832e06098a3e499f82d53853e26235cf P b66c1d3fe3174f64b0936c512ee08f48: stopping tablet replica
I20260925 03:03:20.014228 23300 raft_consensus.cc:2229] T 832e06098a3e499f82d53853e26235cf P b66c1d3fe3174f64b0936c512ee08f48 [term 4 LEADER]: Raft consensus shutting down.
I20260925 03:03:20.015096 23300 raft_consensus.cc:2258] T 832e06098a3e499f82d53853e26235cf P b66c1d3fe3174f64b0936c512ee08f48 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260925 03:03:20.016860 23300 tablet_replica.cc:330] T 138dfa413a144a0e8911a2b402857cb4 P b66c1d3fe3174f64b0936c512ee08f48: stopping tablet replica
I20260925 03:03:20.017343 23300 raft_consensus.cc:2229] T 138dfa413a144a0e8911a2b402857cb4 P b66c1d3fe3174f64b0936c512ee08f48 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260925 03:03:20.017755 23300 raft_consensus.cc:2258] T 138dfa413a144a0e8911a2b402857cb4 P b66c1d3fe3174f64b0936c512ee08f48 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260925 03:03:20.019423 23300 tablet_replica.cc:330] T c30e9a1092fe4d99994480aab8863cf6 P b66c1d3fe3174f64b0936c512ee08f48: stopping tablet replica
I20260925 03:03:20.019887 23300 raft_consensus.cc:2229] T c30e9a1092fe4d99994480aab8863cf6 P b66c1d3fe3174f64b0936c512ee08f48 [term 5 FOLLOWER]: Raft consensus shutting down.
I20260925 03:03:20.020273 23300 raft_consensus.cc:2258] T c30e9a1092fe4d99994480aab8863cf6 P b66c1d3fe3174f64b0936c512ee08f48 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260925 03:03:20.021870 23300 tablet_replica.cc:330] T d01220d66a8c4d34ac86fc3559fa5165 P b66c1d3fe3174f64b0936c512ee08f48: stopping tablet replica
I20260925 03:03:20.022347 23300 raft_consensus.cc:2229] T d01220d66a8c4d34ac86fc3559fa5165 P b66c1d3fe3174f64b0936c512ee08f48 [term 4 LEADER]: Raft consensus shutting down.
I20260925 03:03:20.023078 23300 raft_consensus.cc:2258] T d01220d66a8c4d34ac86fc3559fa5165 P b66c1d3fe3174f64b0936c512ee08f48 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260925 03:03:20.024645 23300 tablet_replica.cc:330] T fb663a8b564c494f98f56845f8a05d6c P b66c1d3fe3174f64b0936c512ee08f48: stopping tablet replica
I20260925 03:03:20.025170 23300 raft_consensus.cc:2229] T fb663a8b564c494f98f56845f8a05d6c P b66c1d3fe3174f64b0936c512ee08f48 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260925 03:03:20.025539 23300 raft_consensus.cc:2258] T fb663a8b564c494f98f56845f8a05d6c P b66c1d3fe3174f64b0936c512ee08f48 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260925 03:03:20.027016 23300 tablet_replica.cc:330] T ef74cafe53e34e1dace8b02d73693521 P b66c1d3fe3174f64b0936c512ee08f48: stopping tablet replica
I20260925 03:03:20.027451 23300 raft_consensus.cc:2229] T ef74cafe53e34e1dace8b02d73693521 P b66c1d3fe3174f64b0936c512ee08f48 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260925 03:03:20.027868 23300 raft_consensus.cc:2258] T ef74cafe53e34e1dace8b02d73693521 P b66c1d3fe3174f64b0936c512ee08f48 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260925 03:03:20.029408 23300 tablet_replica.cc:330] T 2baf62cd63564f4ba08b201e40e19d55 P b66c1d3fe3174f64b0936c512ee08f48: stopping tablet replica
I20260925 03:03:20.029844 23300 raft_consensus.cc:2229] T 2baf62cd63564f4ba08b201e40e19d55 P b66c1d3fe3174f64b0936c512ee08f48 [term 5 FOLLOWER]: Raft consensus shutting down.
I20260925 03:03:20.030246 23300 raft_consensus.cc:2258] T 2baf62cd63564f4ba08b201e40e19d55 P b66c1d3fe3174f64b0936c512ee08f48 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260925 03:03:20.031802 23300 tablet_replica.cc:330] T 43de311d912b475eb01e1f7c9a65412a P b66c1d3fe3174f64b0936c512ee08f48: stopping tablet replica
I20260925 03:03:20.032245 23300 raft_consensus.cc:2229] T 43de311d912b475eb01e1f7c9a65412a P b66c1d3fe3174f64b0936c512ee08f48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 03:03:20.032595 23300 raft_consensus.cc:2258] T 43de311d912b475eb01e1f7c9a65412a P b66c1d3fe3174f64b0936c512ee08f48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 03:03:20.034132 23300 tablet_replica.cc:330] T 9de619025d19432c8df31a19bb6fc446 P b66c1d3fe3174f64b0936c512ee08f48: stopping tablet replica
I20260925 03:03:20.034583 23300 raft_consensus.cc:2229] T 9de619025d19432c8df31a19bb6fc446 P b66c1d3fe3174f64b0936c512ee08f48 [term 2 LEADER]: Raft consensus shutting down.
I20260925 03:03:20.035306 23300 raft_consensus.cc:2258] T 9de619025d19432c8df31a19bb6fc446 P b66c1d3fe3174f64b0936c512ee08f48 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260925 03:03:20.062709 23300 tablet_server.cc:194] TabletServer@127.22.193.3:0 shutdown complete.
I20260925 03:03:20.078969 23300 master.cc:537] Master@127.22.193.62:42563 shutting down...
I20260925 03:03:20.337148 23300 raft_consensus.cc:2229] T 00000000000000000000000000000000 P d0542f50245c40929690275ab889db50 [term 1 LEADER]: Raft consensus shutting down.
I20260925 03:03:20.337782 23300 raft_consensus.cc:2258] T 00000000000000000000000000000000 P d0542f50245c40929690275ab889db50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 03:03:20.338213 23300 tablet_replica.cc:330] T 00000000000000000000000000000000 P d0542f50245c40929690275ab889db50: stopping tablet replica
I20260925 03:03:20.374607 23300 master.cc:552] Master@127.22.193.62:42563 shutdown complete.
2edab657a33a4a0abfbd3262b1668483 4
b66c1d3fe3174f64b0936c512ee08f48 1
fa551b23283b4034b256ba25acec541b 4
2edab657a33a4a0abfbd3262b1668483 3
b66c1d3fe3174f64b0936c512ee08f48 3
fa551b23283b4034b256ba25acec541b 3
2edab657a33a4a0abfbd3262b1668483 8
b66c1d3fe3174f64b0936c512ee08f48 1
fa551b23283b4034b256ba25acec541b 0
2edab657a33a4a0abfbd3262b1668483 3
b66c1d3fe3174f64b0936c512ee08f48 3
fa551b23283b4034b256ba25acec541b 3
[ OK ] LeaderRebalancerTest.FunctionalTestForDivided (32084 ms)
[ RUN ] LeaderRebalancerTest.FunctionalTestForNotDivided
I20260925 03:03:20.413239 23300 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.22.193.62:33513
I20260925 03:03:20.414305 23300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 03:03:20.419818 23822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:03:20.420138 23823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:03:20.421175 23825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 03:03:20.421677 23300 server_base.cc:1034] running on GCE node
I20260925 03:03:20.422607 23300 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260925 03:03:20.422792 23300 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260925 03:03:20.422901 23300 hybrid_clock.cc:648] HybridClock initialized: now 1790305400422893 us; error 0 us; skew 500 ppm
I20260925 03:03:20.423342 23300 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 03:03:20.432024 23300 webserver.cc:460] Webserver started at http://127.22.193.62:34581/ using document root <none> and password file <none>
I20260925 03:03:20.432504 23300 fs_manager.cc:359] Metadata directory not provided
I20260925 03:03:20.432655 23300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 03:03:20.432964 23300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 03:03:20.434126 23300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklAj8hX/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1790305368086980-23300-0/minicluster-data/master-0-root/instance:
uuid: "06b4bc18a82b4b8fb4a33975c07d3a12"
format_stamp: "Formatted at 2026-09-25 03:03:20 on dist-test-slave-cswq"
I20260925 03:03:20.438768 23300 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 03:03:20.442040 23830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 03:03:20.442874 23300 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 03:03:20.443146 23300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklAj8hX/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1790305368086980-23300-0/minicluster-data/master-0-root
uuid: "06b4bc18a82b4b8fb4a33975c07d3a12"
format_stamp: "Formatted at 2026-09-25 03:03:20 on dist-test-slave-cswq"
I20260925 03:03:20.443444 23300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklAj8hX/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1790305368086980-23300-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-tasklAj8hX/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1790305368086980-23300-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-tasklAj8hX/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1790305368086980-23300-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 03:03:20.464159 23300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 03:03:20.465512 23300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 03:03:20.505532 23300 rpc_server.cc:307] RPC server started. Bound to: 127.22.193.62:33513
I20260925 03:03:20.505625 23881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.193.62:33513 every 8 connection(s)
I20260925 03:03:20.509442 23882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:20.519443 23882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06b4bc18a82b4b8fb4a33975c07d3a12: Bootstrap starting.
I20260925 03:03:20.524259 23882 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 06b4bc18a82b4b8fb4a33975c07d3a12: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:20.528867 23882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06b4bc18a82b4b8fb4a33975c07d3a12: No bootstrap required, opened a new log
I20260925 03:03:20.530952 23882 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06b4bc18a82b4b8fb4a33975c07d3a12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06b4bc18a82b4b8fb4a33975c07d3a12" member_type: VOTER }
I20260925 03:03:20.531446 23882 raft_consensus.cc:374] T 00000000000000000000000000000000 P 06b4bc18a82b4b8fb4a33975c07d3a12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:20.531678 23882 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06b4bc18a82b4b8fb4a33975c07d3a12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06b4bc18a82b4b8fb4a33975c07d3a12, State: Initialized, Role: FOLLOWER
I20260925 03:03:20.532239 23882 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06b4bc18a82b4b8fb4a33975c07d3a12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06b4bc18a82b4b8fb4a33975c07d3a12" member_type: VOTER }
I20260925 03:03:20.532680 23882 raft_consensus.cc:388] T 00000000000000000000000000000000 P 06b4bc18a82b4b8fb4a33975c07d3a12 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 03:03:20.532943 23882 raft_consensus.cc:482] T 00000000000000000000000000000000 P 06b4bc18a82b4b8fb4a33975c07d3a12 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 03:03:20.533197 23882 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06b4bc18a82b4b8fb4a33975c07d3a12 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:20.537169 23882 raft_consensus.cc:504] T 00000000000000000000000000000000 P 06b4bc18a82b4b8fb4a33975c07d3a12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06b4bc18a82b4b8fb4a33975c07d3a12" member_type: VOTER }
I20260925 03:03:20.537693 23882 leader_election.cc:302] T 00000000000000000000000000000000 P 06b4bc18a82b4b8fb4a33975c07d3a12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 06b4bc18a82b4b8fb4a33975c07d3a12; no voters:
I20260925 03:03:20.538829 23882 leader_election.cc:288] T 00000000000000000000000000000000 P 06b4bc18a82b4b8fb4a33975c07d3a12 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 03:03:20.539320 23885 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 06b4bc18a82b4b8fb4a33975c07d3a12 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:03:20.540724 23885 raft_consensus.cc:686] T 00000000000000000000000000000000 P 06b4bc18a82b4b8fb4a33975c07d3a12 [term 1 LEADER]: Becoming Leader. State: Replica: 06b4bc18a82b4b8fb4a33975c07d3a12, State: Running, Role: LEADER
I20260925 03:03:20.541399 23885 consensus_queue.cc:237] T 00000000000000000000000000000000 P 06b4bc18a82b4b8fb4a33975c07d3a12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06b4bc18a82b4b8fb4a33975c07d3a12" member_type: VOTER }
I20260925 03:03:20.541752 23882 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06b4bc18a82b4b8fb4a33975c07d3a12 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 03:03:20.545841 23886 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06b4bc18a82b4b8fb4a33975c07d3a12 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "06b4bc18a82b4b8fb4a33975c07d3a12" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06b4bc18a82b4b8fb4a33975c07d3a12" member_type: VOTER } }
I20260925 03:03:20.546622 23886 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06b4bc18a82b4b8fb4a33975c07d3a12 [sys.catalog]: This master's current role is: LEADER
I20260925 03:03:20.546602 23887 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06b4bc18a82b4b8fb4a33975c07d3a12 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06b4bc18a82b4b8fb4a33975c07d3a12. Latest consensus state: current_term: 1 leader_uuid: "06b4bc18a82b4b8fb4a33975c07d3a12" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06b4bc18a82b4b8fb4a33975c07d3a12" member_type: VOTER } }
I20260925 03:03:20.547405 23887 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06b4bc18a82b4b8fb4a33975c07d3a12 [sys.catalog]: This master's current role is: LEADER
I20260925 03:03:20.550519 23893 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 03:03:20.557893 23893 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 03:03:20.560539 23300 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260925 03:03:20.567178 23893 catalog_manager.cc:1347] Generated new cluster ID: 346302f309a84bffa4d215f767e57152
I20260925 03:03:20.567487 23893 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 03:03:20.583163 23893 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 03:03:20.584810 23893 catalog_manager.cc:1504] Loading token signing keys...
I20260925 03:03:20.596516 23893 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 06b4bc18a82b4b8fb4a33975c07d3a12: Generated new TSK 0
I20260925 03:03:20.597182 23893 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 03:03:20.627099 23300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 03:03:20.633068 23903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:03:20.633950 23904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:03:20.635603 23906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 03:03:20.636132 23300 server_base.cc:1034] running on GCE node
I20260925 03:03:20.637051 23300 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260925 03:03:20.637233 23300 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260925 03:03:20.637390 23300 hybrid_clock.cc:648] HybridClock initialized: now 1790305400637380 us; error 0 us; skew 500 ppm
I20260925 03:03:20.637923 23300 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 03:03:20.640336 23300 webserver.cc:460] Webserver started at http://127.22.193.1:36487/ using document root <none> and password file <none>
I20260925 03:03:20.640905 23300 fs_manager.cc:359] Metadata directory not provided
I20260925 03:03:20.641072 23300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 03:03:20.641366 23300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 03:03:20.642479 23300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklAj8hX/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1790305368086980-23300-0/minicluster-data/ts-0-root/instance:
uuid: "e8496e8e22c447f9b270e862bbb272a3"
format_stamp: "Formatted at 2026-09-25 03:03:20 on dist-test-slave-cswq"
I20260925 03:03:20.647132 23300 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 03:03:20.650489 23911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 03:03:20.651216 23300 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 03:03:20.651515 23300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklAj8hX/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1790305368086980-23300-0/minicluster-data/ts-0-root
uuid: "e8496e8e22c447f9b270e862bbb272a3"
format_stamp: "Formatted at 2026-09-25 03:03:20 on dist-test-slave-cswq"
I20260925 03:03:20.651782 23300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklAj8hX/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1790305368086980-23300-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-tasklAj8hX/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1790305368086980-23300-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-tasklAj8hX/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1790305368086980-23300-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 03:03:20.662598 23300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 03:03:20.663758 23300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 03:03:20.665283 23300 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 03:03:20.667728 23300 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 03:03:20.667968 23300 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 03:03:20.668215 23300 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 03:03:20.668380 23300 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 03:03:20.708550 23300 rpc_server.cc:307] RPC server started. Bound to: 127.22.193.1:44975
I20260925 03:03:20.708642 23973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.193.1:44975 every 8 connection(s)
I20260925 03:03:20.712970 23300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 03:03:20.720185 23978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:03:20.721271 23979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 03:03:20.723035 23300 server_base.cc:1034] running on GCE node
W20260925 03:03:20.723673 23981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 03:03:20.724718 23300 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260925 03:03:20.724962 23300 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260925 03:03:20.725144 23300 hybrid_clock.cc:648] HybridClock initialized: now 1790305400725129 us; error 0 us; skew 500 ppm
I20260925 03:03:20.725651 23300 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 03:03:20.727855 23974 heartbeater.cc:347] Connected to a master server at 127.22.193.62:33513
I20260925 03:03:20.728268 23974 heartbeater.cc:464] Registering TS with master...
I20260925 03:03:20.728543 23300 webserver.cc:460] Webserver started at http://127.22.193.2:41319/ using document root <none> and password file <none>
I20260925 03:03:20.729107 23974 heartbeater.cc:511] Master 127.22.193.62:33513 requested a full tablet report, sending...
I20260925 03:03:20.729206 23300 fs_manager.cc:359] Metadata directory not provided
I20260925 03:03:20.729506 23300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 03:03:20.729818 23300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 03:03:20.731139 23300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklAj8hX/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1790305368086980-23300-0/minicluster-data/ts-1-root/instance:
uuid: "1c37142ee9c043d7a5aca01030e1bbf5"
format_stamp: "Formatted at 2026-09-25 03:03:20 on dist-test-slave-cswq"
I20260925 03:03:20.731298 23847 ts_manager.cc:194] Registered new tserver with Master: e8496e8e22c447f9b270e862bbb272a3 (127.22.193.1:44975)
I20260925 03:03:20.733238 23847 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44528
I20260925 03:03:20.735924 23300 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 03:03:20.739315 23986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 03:03:20.740080 23300 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 03:03:20.740398 23300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklAj8hX/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1790305368086980-23300-0/minicluster-data/ts-1-root
uuid: "1c37142ee9c043d7a5aca01030e1bbf5"
format_stamp: "Formatted at 2026-09-25 03:03:20 on dist-test-slave-cswq"
I20260925 03:03:20.740741 23300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklAj8hX/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1790305368086980-23300-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-tasklAj8hX/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1790305368086980-23300-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-tasklAj8hX/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1790305368086980-23300-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 03:03:20.761335 23300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 03:03:20.762513 23300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 03:03:20.763960 23300 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 03:03:20.766206 23300 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 03:03:20.766395 23300 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 03:03:20.766621 23300 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 03:03:20.766768 23300 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 03:03:20.805611 23300 rpc_server.cc:307] RPC server started. Bound to: 127.22.193.2:40513
I20260925 03:03:20.805752 24048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.193.2:40513 every 8 connection(s)
I20260925 03:03:20.810456 23300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 03:03:20.818256 24052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:03:20.818584 24053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:03:20.821950 24055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 03:03:20.822472 23300 server_base.cc:1034] running on GCE node
I20260925 03:03:20.823599 23300 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260925 03:03:20.823796 23300 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260925 03:03:20.824015 23300 hybrid_clock.cc:648] HybridClock initialized: now 1790305400823996 us; error 0 us; skew 500 ppm
I20260925 03:03:20.824067 24049 heartbeater.cc:347] Connected to a master server at 127.22.193.62:33513
I20260925 03:03:20.824458 24049 heartbeater.cc:464] Registering TS with master...
I20260925 03:03:20.824606 23300 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 03:03:20.825263 24049 heartbeater.cc:511] Master 127.22.193.62:33513 requested a full tablet report, sending...
I20260925 03:03:20.827450 23847 ts_manager.cc:194] Registered new tserver with Master: 1c37142ee9c043d7a5aca01030e1bbf5 (127.22.193.2:40513)
I20260925 03:03:20.827554 23300 webserver.cc:460] Webserver started at http://127.22.193.3:38035/ using document root <none> and password file <none>
I20260925 03:03:20.828101 23300 fs_manager.cc:359] Metadata directory not provided
I20260925 03:03:20.828285 23300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 03:03:20.828527 23300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 03:03:20.829063 23847 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44542
I20260925 03:03:20.829735 23300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklAj8hX/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1790305368086980-23300-0/minicluster-data/ts-2-root/instance:
uuid: "5dae02dd94274f54ab9c52ab57a880b0"
format_stamp: "Formatted at 2026-09-25 03:03:20 on dist-test-slave-cswq"
I20260925 03:03:20.834322 23300 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 03:03:20.837689 24060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 03:03:20.838614 23300 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 03:03:20.838914 23300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklAj8hX/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1790305368086980-23300-0/minicluster-data/ts-2-root
uuid: "5dae02dd94274f54ab9c52ab57a880b0"
format_stamp: "Formatted at 2026-09-25 03:03:20 on dist-test-slave-cswq"
I20260925 03:03:20.839181 23300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklAj8hX/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1790305368086980-23300-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-tasklAj8hX/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1790305368086980-23300-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-tasklAj8hX/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1790305368086980-23300-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 03:03:20.875705 23300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 03:03:20.876951 23300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 03:03:20.878417 23300 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 03:03:20.880591 23300 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 03:03:20.880826 23300 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 03:03:20.881049 23300 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 03:03:20.881206 23300 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 03:03:20.919834 23300 rpc_server.cc:307] RPC server started. Bound to: 127.22.193.3:46383
I20260925 03:03:20.919930 24122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.193.3:46383 every 8 connection(s)
I20260925 03:03:20.934662 24123 heartbeater.cc:347] Connected to a master server at 127.22.193.62:33513
I20260925 03:03:20.935052 24123 heartbeater.cc:464] Registering TS with master...
I20260925 03:03:20.935775 24123 heartbeater.cc:511] Master 127.22.193.62:33513 requested a full tablet report, sending...
I20260925 03:03:20.937742 23847 ts_manager.cc:194] Registered new tserver with Master: 5dae02dd94274f54ab9c52ab57a880b0 (127.22.193.3:46383)
I20260925 03:03:20.937894 23300 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.014899731s
I20260925 03:03:20.940061 23847 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44546
I20260925 03:03:20.940899 23300 test_util.cc:268] Using random seed: -176789174
I20260925 03:03:20.966660 23847 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44548:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s""\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 03:03:20.969624 23847 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 03:03:21.054113 24014 tablet_service.cc:1467] Processing CreateTablet for tablet f58a83ec6f4149108eb0567cd7817ab5 (DEFAULT_TABLE table=test-workload [id=b40c4b0a7da94c2499e90ca3f3ed657d]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260925 03:03:21.054489 24013 tablet_service.cc:1467] Processing CreateTablet for tablet 7f1e503d876c45fa853edeaf3b836180 (DEFAULT_TABLE table=test-workload [id=b40c4b0a7da94c2499e90ca3f3ed657d]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260925 03:03:21.055613 24014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f58a83ec6f4149108eb0567cd7817ab5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:21.056221 24013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f1e503d876c45fa853edeaf3b836180. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:21.057315 24006 tablet_service.cc:1467] Processing CreateTablet for tablet be306de199ff4dfb976075219c02ea8e (DEFAULT_TABLE table=test-workload [id=b40c4b0a7da94c2499e90ca3f3ed657d]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260925 03:03:21.057514 24012 tablet_service.cc:1467] Processing CreateTablet for tablet c9f3dfa77c7845a49f80417befebb978 (DEFAULT_TABLE table=test-workload [id=b40c4b0a7da94c2499e90ca3f3ed657d]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260925 03:03:21.058677 24006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be306de199ff4dfb976075219c02ea8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:21.059453 24012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9f3dfa77c7845a49f80417befebb978. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:21.059015 24011 tablet_service.cc:1467] Processing CreateTablet for tablet 48c4fddba997473f8a501507277b8860 (DEFAULT_TABLE table=test-workload [id=b40c4b0a7da94c2499e90ca3f3ed657d]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260925 03:03:21.061884 24009 tablet_service.cc:1467] Processing CreateTablet for tablet 400a066077234276876ac579e8353ad2 (DEFAULT_TABLE table=test-workload [id=b40c4b0a7da94c2499e90ca3f3ed657d]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260925 03:03:21.063279 24009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 400a066077234276876ac579e8353ad2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:21.065831 24008 tablet_service.cc:1467] Processing CreateTablet for tablet 78943162e8394d039f00afadd62893f3 (DEFAULT_TABLE table=test-workload [id=b40c4b0a7da94c2499e90ca3f3ed657d]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260925 03:03:21.067107 24008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78943162e8394d039f00afadd62893f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:21.069917 24007 tablet_service.cc:1467] Processing CreateTablet for tablet 5055d207a0994afbbbc25a2c39455f87 (DEFAULT_TABLE table=test-workload [id=b40c4b0a7da94c2499e90ca3f3ed657d]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260925 03:03:21.070341 24005 tablet_service.cc:1467] Processing CreateTablet for tablet 5ac7083552944834bb61700cfbe60faa (DEFAULT_TABLE table=test-workload [id=b40c4b0a7da94c2499e90ca3f3ed657d]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260925 03:03:21.071238 24007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5055d207a0994afbbbc25a2c39455f87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:21.071830 24005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ac7083552944834bb61700cfbe60faa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:21.073339 24010 tablet_service.cc:1467] Processing CreateTablet for tablet 845902c057d54c5ab9ddc861f622125c (DEFAULT_TABLE table=test-workload [id=b40c4b0a7da94c2499e90ca3f3ed657d]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260925 03:03:21.074632 24010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 845902c057d54c5ab9ddc861f622125c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:21.074484 23938 tablet_service.cc:1467] Processing CreateTablet for tablet 7f1e503d876c45fa853edeaf3b836180 (DEFAULT_TABLE table=test-workload [id=b40c4b0a7da94c2499e90ca3f3ed657d]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260925 03:03:21.075066 23937 tablet_service.cc:1467] Processing CreateTablet for tablet c9f3dfa77c7845a49f80417befebb978 (DEFAULT_TABLE table=test-workload [id=b40c4b0a7da94c2499e90ca3f3ed657d]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260925 03:03:21.075731 23938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f1e503d876c45fa853edeaf3b836180. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:21.076295 23937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9f3dfa77c7845a49f80417befebb978. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:21.062949 23939 tablet_service.cc:1467] Processing CreateTablet for tablet f58a83ec6f4149108eb0567cd7817ab5 (DEFAULT_TABLE table=test-workload [id=b40c4b0a7da94c2499e90ca3f3ed657d]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260925 03:03:21.079965 23936 tablet_service.cc:1467] Processing CreateTablet for tablet 48c4fddba997473f8a501507277b8860 (DEFAULT_TABLE table=test-workload [id=b40c4b0a7da94c2499e90ca3f3ed657d]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260925 03:03:21.080415 23939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f58a83ec6f4149108eb0567cd7817ab5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:21.081262 23936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48c4fddba997473f8a501507277b8860. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:21.081861 24011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48c4fddba997473f8a501507277b8860. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:21.094745 23935 tablet_service.cc:1467] Processing CreateTablet for tablet 845902c057d54c5ab9ddc861f622125c (DEFAULT_TABLE table=test-workload [id=b40c4b0a7da94c2499e90ca3f3ed657d]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260925 03:03:21.096135 23935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 845902c057d54c5ab9ddc861f622125c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:21.097971 23934 tablet_service.cc:1467] Processing CreateTablet for tablet 400a066077234276876ac579e8353ad2 (DEFAULT_TABLE table=test-workload [id=b40c4b0a7da94c2499e90ca3f3ed657d]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260925 03:03:21.099364 23934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 400a066077234276876ac579e8353ad2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:21.104873 24139 tablet_bootstrap.cc:492] T c9f3dfa77c7845a49f80417befebb978 P 1c37142ee9c043d7a5aca01030e1bbf5: Bootstrap starting.
I20260925 03:03:21.107693 24138 tablet_bootstrap.cc:492] T 7f1e503d876c45fa853edeaf3b836180 P e8496e8e22c447f9b270e862bbb272a3: Bootstrap starting.
I20260925 03:03:21.112857 24139 tablet_bootstrap.cc:654] T c9f3dfa77c7845a49f80417befebb978 P 1c37142ee9c043d7a5aca01030e1bbf5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:21.114157 24138 tablet_bootstrap.cc:654] T 7f1e503d876c45fa853edeaf3b836180 P e8496e8e22c447f9b270e862bbb272a3: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:21.116575 23933 tablet_service.cc:1467] Processing CreateTablet for tablet 78943162e8394d039f00afadd62893f3 (DEFAULT_TABLE table=test-workload [id=b40c4b0a7da94c2499e90ca3f3ed657d]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260925 03:03:21.118078 23933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78943162e8394d039f00afadd62893f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:21.122754 23932 tablet_service.cc:1467] Processing CreateTablet for tablet be306de199ff4dfb976075219c02ea8e (DEFAULT_TABLE table=test-workload [id=b40c4b0a7da94c2499e90ca3f3ed657d]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260925 03:03:21.123404 23937 tablet_service.cc:1467] Processing CreateTablet for tablet 5055d207a0994afbbbc25a2c39455f87 (DEFAULT_TABLE table=test-workload [id=b40c4b0a7da94c2499e90ca3f3ed657d]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260925 03:03:21.124104 23932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be306de199ff4dfb976075219c02ea8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:21.124826 23937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5055d207a0994afbbbc25a2c39455f87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:21.126117 23931 tablet_service.cc:1467] Processing CreateTablet for tablet 5ac7083552944834bb61700cfbe60faa (DEFAULT_TABLE table=test-workload [id=b40c4b0a7da94c2499e90ca3f3ed657d]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260925 03:03:21.127393 23931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ac7083552944834bb61700cfbe60faa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:21.169135 24088 tablet_service.cc:1467] Processing CreateTablet for tablet f58a83ec6f4149108eb0567cd7817ab5 (DEFAULT_TABLE table=test-workload [id=b40c4b0a7da94c2499e90ca3f3ed657d]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260925 03:03:21.169993 24087 tablet_service.cc:1467] Processing CreateTablet for tablet 7f1e503d876c45fa853edeaf3b836180 (DEFAULT_TABLE table=test-workload [id=b40c4b0a7da94c2499e90ca3f3ed657d]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260925 03:03:21.170511 24088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f58a83ec6f4149108eb0567cd7817ab5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:21.172506 24086 tablet_service.cc:1467] Processing CreateTablet for tablet c9f3dfa77c7845a49f80417befebb978 (DEFAULT_TABLE table=test-workload [id=b40c4b0a7da94c2499e90ca3f3ed657d]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260925 03:03:21.177014 24087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f1e503d876c45fa853edeaf3b836180. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:21.177775 24085 tablet_service.cc:1467] Processing CreateTablet for tablet 48c4fddba997473f8a501507277b8860 (DEFAULT_TABLE table=test-workload [id=b40c4b0a7da94c2499e90ca3f3ed657d]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260925 03:03:21.178495 24083 tablet_service.cc:1467] Processing CreateTablet for tablet 400a066077234276876ac579e8353ad2 (DEFAULT_TABLE table=test-workload [id=b40c4b0a7da94c2499e90ca3f3ed657d]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260925 03:03:21.179147 24086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9f3dfa77c7845a49f80417befebb978. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:21.180310 24084 tablet_service.cc:1467] Processing CreateTablet for tablet 845902c057d54c5ab9ddc861f622125c (DEFAULT_TABLE table=test-workload [id=b40c4b0a7da94c2499e90ca3f3ed657d]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260925 03:03:21.181447 24082 tablet_service.cc:1467] Processing CreateTablet for tablet 78943162e8394d039f00afadd62893f3 (DEFAULT_TABLE table=test-workload [id=b40c4b0a7da94c2499e90ca3f3ed657d]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260925 03:03:21.181741 24080 tablet_service.cc:1467] Processing CreateTablet for tablet be306de199ff4dfb976075219c02ea8e (DEFAULT_TABLE table=test-workload [id=b40c4b0a7da94c2499e90ca3f3ed657d]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260925 03:03:21.184465 24079 tablet_service.cc:1467] Processing CreateTablet for tablet 5ac7083552944834bb61700cfbe60faa (DEFAULT_TABLE table=test-workload [id=b40c4b0a7da94c2499e90ca3f3ed657d]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260925 03:03:21.185248 24081 tablet_service.cc:1467] Processing CreateTablet for tablet 5055d207a0994afbbbc25a2c39455f87 (DEFAULT_TABLE table=test-workload [id=b40c4b0a7da94c2499e90ca3f3ed657d]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260925 03:03:21.185977 24085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48c4fddba997473f8a501507277b8860. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:21.186685 24081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5055d207a0994afbbbc25a2c39455f87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:21.188160 24083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 400a066077234276876ac579e8353ad2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:21.189007 24079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ac7083552944834bb61700cfbe60faa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:21.190232 24084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 845902c057d54c5ab9ddc861f622125c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:21.193003 24080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be306de199ff4dfb976075219c02ea8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:21.194087 24082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78943162e8394d039f00afadd62893f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:21.215233 24138 tablet_bootstrap.cc:492] T 7f1e503d876c45fa853edeaf3b836180 P e8496e8e22c447f9b270e862bbb272a3: No bootstrap required, opened a new log
I20260925 03:03:21.215798 24138 ts_tablet_manager.cc:1397] T 7f1e503d876c45fa853edeaf3b836180 P e8496e8e22c447f9b270e862bbb272a3: Time spent bootstrapping tablet: real 0.108s user 0.012s sys 0.000s
I20260925 03:03:21.221504 24138 raft_consensus.cc:348] T 7f1e503d876c45fa853edeaf3b836180 P e8496e8e22c447f9b270e862bbb272a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:21.222514 24138 raft_consensus.cc:374] T 7f1e503d876c45fa853edeaf3b836180 P e8496e8e22c447f9b270e862bbb272a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:21.222859 24138 raft_consensus.cc:729] T 7f1e503d876c45fa853edeaf3b836180 P e8496e8e22c447f9b270e862bbb272a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8496e8e22c447f9b270e862bbb272a3, State: Initialized, Role: FOLLOWER
I20260925 03:03:21.229009 24138 consensus_queue.cc:260] T 7f1e503d876c45fa853edeaf3b836180 P e8496e8e22c447f9b270e862bbb272a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:21.232941 23974 heartbeater.cc:503] Master 127.22.193.62:33513 was elected leader, sending a full tablet report...
I20260925 03:03:21.235647 24139 tablet_bootstrap.cc:492] T c9f3dfa77c7845a49f80417befebb978 P 1c37142ee9c043d7a5aca01030e1bbf5: No bootstrap required, opened a new log
I20260925 03:03:21.236001 24139 ts_tablet_manager.cc:1397] T c9f3dfa77c7845a49f80417befebb978 P 1c37142ee9c043d7a5aca01030e1bbf5: Time spent bootstrapping tablet: real 0.131s user 0.025s sys 0.023s
I20260925 03:03:21.237610 24142 tablet_bootstrap.cc:492] T 5ac7083552944834bb61700cfbe60faa P 5dae02dd94274f54ab9c52ab57a880b0: Bootstrap starting.
I20260925 03:03:21.239691 24138 ts_tablet_manager.cc:1428] T 7f1e503d876c45fa853edeaf3b836180 P e8496e8e22c447f9b270e862bbb272a3: Time spent starting tablet: real 0.024s user 0.007s sys 0.000s
I20260925 03:03:21.240744 24138 tablet_bootstrap.cc:492] T c9f3dfa77c7845a49f80417befebb978 P e8496e8e22c447f9b270e862bbb272a3: Bootstrap starting.
I20260925 03:03:21.244352 24139 raft_consensus.cc:348] T c9f3dfa77c7845a49f80417befebb978 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:21.245124 24139 raft_consensus.cc:374] T c9f3dfa77c7845a49f80417befebb978 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:21.245416 24139 raft_consensus.cc:729] T c9f3dfa77c7845a49f80417befebb978 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c37142ee9c043d7a5aca01030e1bbf5, State: Initialized, Role: FOLLOWER
I20260925 03:03:21.246179 24139 consensus_queue.cc:260] T c9f3dfa77c7845a49f80417befebb978 P 1c37142ee9c043d7a5aca01030e1bbf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:21.254874 24049 heartbeater.cc:503] Master 127.22.193.62:33513 was elected leader, sending a full tablet report...
I20260925 03:03:21.255909 24138 tablet_bootstrap.cc:654] T c9f3dfa77c7845a49f80417befebb978 P e8496e8e22c447f9b270e862bbb272a3: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:21.257381 24139 ts_tablet_manager.cc:1428] T c9f3dfa77c7845a49f80417befebb978 P 1c37142ee9c043d7a5aca01030e1bbf5: Time spent starting tablet: real 0.021s user 0.009s sys 0.005s
I20260925 03:03:21.258348 24139 tablet_bootstrap.cc:492] T 7f1e503d876c45fa853edeaf3b836180 P 1c37142ee9c043d7a5aca01030e1bbf5: Bootstrap starting.
I20260925 03:03:21.261094 24142 tablet_bootstrap.cc:654] T 5ac7083552944834bb61700cfbe60faa P 5dae02dd94274f54ab9c52ab57a880b0: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:21.263777 24139 tablet_bootstrap.cc:654] T 7f1e503d876c45fa853edeaf3b836180 P 1c37142ee9c043d7a5aca01030e1bbf5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:21.268321 24138 tablet_bootstrap.cc:492] T c9f3dfa77c7845a49f80417befebb978 P e8496e8e22c447f9b270e862bbb272a3: No bootstrap required, opened a new log
I20260925 03:03:21.268901 24138 ts_tablet_manager.cc:1397] T c9f3dfa77c7845a49f80417befebb978 P e8496e8e22c447f9b270e862bbb272a3: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.013s
I20260925 03:03:21.271508 24138 raft_consensus.cc:348] T c9f3dfa77c7845a49f80417befebb978 P e8496e8e22c447f9b270e862bbb272a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:21.272384 24138 raft_consensus.cc:374] T c9f3dfa77c7845a49f80417befebb978 P e8496e8e22c447f9b270e862bbb272a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:21.272730 24138 raft_consensus.cc:729] T c9f3dfa77c7845a49f80417befebb978 P e8496e8e22c447f9b270e862bbb272a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8496e8e22c447f9b270e862bbb272a3, State: Initialized, Role: FOLLOWER
I20260925 03:03:21.273495 24138 consensus_queue.cc:260] T c9f3dfa77c7845a49f80417befebb978 P e8496e8e22c447f9b270e862bbb272a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:21.275530 24138 ts_tablet_manager.cc:1428] T c9f3dfa77c7845a49f80417befebb978 P e8496e8e22c447f9b270e862bbb272a3: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260925 03:03:21.277603 24138 tablet_bootstrap.cc:492] T 48c4fddba997473f8a501507277b8860 P e8496e8e22c447f9b270e862bbb272a3: Bootstrap starting.
I20260925 03:03:21.281013 24142 tablet_bootstrap.cc:492] T 5ac7083552944834bb61700cfbe60faa P 5dae02dd94274f54ab9c52ab57a880b0: No bootstrap required, opened a new log
I20260925 03:03:21.281546 24142 ts_tablet_manager.cc:1397] T 5ac7083552944834bb61700cfbe60faa P 5dae02dd94274f54ab9c52ab57a880b0: Time spent bootstrapping tablet: real 0.044s user 0.005s sys 0.018s
I20260925 03:03:21.283883 24138 tablet_bootstrap.cc:654] T 48c4fddba997473f8a501507277b8860 P e8496e8e22c447f9b270e862bbb272a3: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:21.284467 24142 raft_consensus.cc:348] T 5ac7083552944834bb61700cfbe60faa P 5dae02dd94274f54ab9c52ab57a880b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:21.285326 24142 raft_consensus.cc:374] T 5ac7083552944834bb61700cfbe60faa P 5dae02dd94274f54ab9c52ab57a880b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:21.285614 24142 raft_consensus.cc:729] T 5ac7083552944834bb61700cfbe60faa P 5dae02dd94274f54ab9c52ab57a880b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dae02dd94274f54ab9c52ab57a880b0, State: Initialized, Role: FOLLOWER
I20260925 03:03:21.286487 24142 consensus_queue.cc:260] T 5ac7083552944834bb61700cfbe60faa P 5dae02dd94274f54ab9c52ab57a880b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:21.292124 24123 heartbeater.cc:503] Master 127.22.193.62:33513 was elected leader, sending a full tablet report...
I20260925 03:03:21.293947 24142 ts_tablet_manager.cc:1428] T 5ac7083552944834bb61700cfbe60faa P 5dae02dd94274f54ab9c52ab57a880b0: Time spent starting tablet: real 0.012s user 0.007s sys 0.003s
I20260925 03:03:21.294966 24139 tablet_bootstrap.cc:492] T 7f1e503d876c45fa853edeaf3b836180 P 1c37142ee9c043d7a5aca01030e1bbf5: No bootstrap required, opened a new log
I20260925 03:03:21.295138 24142 tablet_bootstrap.cc:492] T 5055d207a0994afbbbc25a2c39455f87 P 5dae02dd94274f54ab9c52ab57a880b0: Bootstrap starting.
I20260925 03:03:21.295553 24139 ts_tablet_manager.cc:1397] T 7f1e503d876c45fa853edeaf3b836180 P 1c37142ee9c043d7a5aca01030e1bbf5: Time spent bootstrapping tablet: real 0.037s user 0.010s sys 0.024s
I20260925 03:03:21.298427 24139 raft_consensus.cc:348] T 7f1e503d876c45fa853edeaf3b836180 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:21.299228 24139 raft_consensus.cc:374] T 7f1e503d876c45fa853edeaf3b836180 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:21.299513 24139 raft_consensus.cc:729] T 7f1e503d876c45fa853edeaf3b836180 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c37142ee9c043d7a5aca01030e1bbf5, State: Initialized, Role: FOLLOWER
I20260925 03:03:21.300169 24139 consensus_queue.cc:260] T 7f1e503d876c45fa853edeaf3b836180 P 1c37142ee9c043d7a5aca01030e1bbf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:21.302075 24139 ts_tablet_manager.cc:1428] T 7f1e503d876c45fa853edeaf3b836180 P 1c37142ee9c043d7a5aca01030e1bbf5: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260925 03:03:21.302121 24138 tablet_bootstrap.cc:492] T 48c4fddba997473f8a501507277b8860 P e8496e8e22c447f9b270e862bbb272a3: No bootstrap required, opened a new log
I20260925 03:03:21.302697 24138 ts_tablet_manager.cc:1397] T 48c4fddba997473f8a501507277b8860 P e8496e8e22c447f9b270e862bbb272a3: Time spent bootstrapping tablet: real 0.025s user 0.014s sys 0.009s
I20260925 03:03:21.302999 24139 tablet_bootstrap.cc:492] T 5ac7083552944834bb61700cfbe60faa P 1c37142ee9c043d7a5aca01030e1bbf5: Bootstrap starting.
I20260925 03:03:21.304958 24144 raft_consensus.cc:482] T c9f3dfa77c7845a49f80417befebb978 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:21.305395 24138 raft_consensus.cc:348] T 48c4fddba997473f8a501507277b8860 P e8496e8e22c447f9b270e862bbb272a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:21.305434 24144 raft_consensus.cc:504] T c9f3dfa77c7845a49f80417befebb978 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:21.308445 24142 tablet_bootstrap.cc:654] T 5055d207a0994afbbbc25a2c39455f87 P 5dae02dd94274f54ab9c52ab57a880b0: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:21.308835 24139 tablet_bootstrap.cc:654] T 5ac7083552944834bb61700cfbe60faa P 1c37142ee9c043d7a5aca01030e1bbf5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:21.327967 24152 raft_consensus.cc:482] T 7f1e503d876c45fa853edeaf3b836180 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:21.328722 24144 leader_election.cc:288] T c9f3dfa77c7845a49f80417befebb978 P 1c37142ee9c043d7a5aca01030e1bbf5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e8496e8e22c447f9b270e862bbb272a3 (127.22.193.1:44975), 5dae02dd94274f54ab9c52ab57a880b0 (127.22.193.3:46383)
I20260925 03:03:21.328557 24152 raft_consensus.cc:504] T 7f1e503d876c45fa853edeaf3b836180 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:21.333154 23949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9f3dfa77c7845a49f80417befebb978" candidate_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8496e8e22c447f9b270e862bbb272a3" is_pre_election: true
I20260925 03:03:21.333873 24138 raft_consensus.cc:374] T 48c4fddba997473f8a501507277b8860 P e8496e8e22c447f9b270e862bbb272a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:21.333959 23949 raft_consensus.cc:2454] T c9f3dfa77c7845a49f80417befebb978 P e8496e8e22c447f9b270e862bbb272a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c37142ee9c043d7a5aca01030e1bbf5 in term 0.
I20260925 03:03:21.334779 23948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f1e503d876c45fa853edeaf3b836180" candidate_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8496e8e22c447f9b270e862bbb272a3" is_pre_election: true
I20260925 03:03:21.335434 23948 raft_consensus.cc:2454] T 7f1e503d876c45fa853edeaf3b836180 P e8496e8e22c447f9b270e862bbb272a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c37142ee9c043d7a5aca01030e1bbf5 in term 0.
I20260925 03:03:21.335458 24152 leader_election.cc:288] T 7f1e503d876c45fa853edeaf3b836180 P 1c37142ee9c043d7a5aca01030e1bbf5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e8496e8e22c447f9b270e862bbb272a3 (127.22.193.1:44975), 5dae02dd94274f54ab9c52ab57a880b0 (127.22.193.3:46383)
I20260925 03:03:21.335374 23988 leader_election.cc:302] T c9f3dfa77c7845a49f80417befebb978 P 1c37142ee9c043d7a5aca01030e1bbf5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c37142ee9c043d7a5aca01030e1bbf5, e8496e8e22c447f9b270e862bbb272a3; no voters:
I20260925 03:03:21.336573 24144 raft_consensus.cc:2781] T c9f3dfa77c7845a49f80417befebb978 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 03:03:21.336730 23988 leader_election.cc:302] T 7f1e503d876c45fa853edeaf3b836180 P 1c37142ee9c043d7a5aca01030e1bbf5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c37142ee9c043d7a5aca01030e1bbf5, e8496e8e22c447f9b270e862bbb272a3; no voters:
I20260925 03:03:21.337049 24144 raft_consensus.cc:482] T c9f3dfa77c7845a49f80417befebb978 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:03:21.337621 24144 raft_consensus.cc:3037] T c9f3dfa77c7845a49f80417befebb978 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:21.343292 24152 raft_consensus.cc:2781] T 7f1e503d876c45fa853edeaf3b836180 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 03:03:21.344547 24152 raft_consensus.cc:482] T 7f1e503d876c45fa853edeaf3b836180 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:03:21.346012 24138 raft_consensus.cc:729] T 48c4fddba997473f8a501507277b8860 P e8496e8e22c447f9b270e862bbb272a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8496e8e22c447f9b270e862bbb272a3, State: Initialized, Role: FOLLOWER
I20260925 03:03:21.345660 24144 raft_consensus.cc:504] T c9f3dfa77c7845a49f80417befebb978 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:21.347999 24144 leader_election.cc:288] T c9f3dfa77c7845a49f80417befebb978 P 1c37142ee9c043d7a5aca01030e1bbf5 [CANDIDATE]: Term 1 election: Requested vote from peers e8496e8e22c447f9b270e862bbb272a3 (127.22.193.1:44975), 5dae02dd94274f54ab9c52ab57a880b0 (127.22.193.3:46383)
I20260925 03:03:21.349607 24139 tablet_bootstrap.cc:492] T 5ac7083552944834bb61700cfbe60faa P 1c37142ee9c043d7a5aca01030e1bbf5: No bootstrap required, opened a new log
I20260925 03:03:21.345160 24152 raft_consensus.cc:3037] T 7f1e503d876c45fa853edeaf3b836180 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:21.350090 24139 ts_tablet_manager.cc:1397] T 5ac7083552944834bb61700cfbe60faa P 1c37142ee9c043d7a5aca01030e1bbf5: Time spent bootstrapping tablet: real 0.047s user 0.019s sys 0.019s
I20260925 03:03:21.351152 24142 tablet_bootstrap.cc:492] T 5055d207a0994afbbbc25a2c39455f87 P 5dae02dd94274f54ab9c52ab57a880b0: No bootstrap required, opened a new log
I20260925 03:03:21.351624 24142 ts_tablet_manager.cc:1397] T 5055d207a0994afbbbc25a2c39455f87 P 5dae02dd94274f54ab9c52ab57a880b0: Time spent bootstrapping tablet: real 0.057s user 0.032s sys 0.014s
I20260925 03:03:21.352145 24139 raft_consensus.cc:348] T 5ac7083552944834bb61700cfbe60faa P 1c37142ee9c043d7a5aca01030e1bbf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:21.346824 24138 consensus_queue.cc:260] T 48c4fddba997473f8a501507277b8860 P e8496e8e22c447f9b270e862bbb272a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:21.352945 24139 raft_consensus.cc:374] T 5ac7083552944834bb61700cfbe60faa P 1c37142ee9c043d7a5aca01030e1bbf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:21.353242 24139 raft_consensus.cc:729] T 5ac7083552944834bb61700cfbe60faa P 1c37142ee9c043d7a5aca01030e1bbf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c37142ee9c043d7a5aca01030e1bbf5, State: Initialized, Role: FOLLOWER
I20260925 03:03:21.354199 24142 raft_consensus.cc:348] T 5055d207a0994afbbbc25a2c39455f87 P 5dae02dd94274f54ab9c52ab57a880b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:21.354914 24142 raft_consensus.cc:374] T 5055d207a0994afbbbc25a2c39455f87 P 5dae02dd94274f54ab9c52ab57a880b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:21.355239 24142 raft_consensus.cc:729] T 5055d207a0994afbbbc25a2c39455f87 P 5dae02dd94274f54ab9c52ab57a880b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dae02dd94274f54ab9c52ab57a880b0, State: Initialized, Role: FOLLOWER
I20260925 03:03:21.355937 24142 consensus_queue.cc:260] T 5055d207a0994afbbbc25a2c39455f87 P 5dae02dd94274f54ab9c52ab57a880b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:21.354100 24139 consensus_queue.cc:260] T 5ac7083552944834bb61700cfbe60faa P 1c37142ee9c043d7a5aca01030e1bbf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:21.357358 23948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9f3dfa77c7845a49f80417befebb978" candidate_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8496e8e22c447f9b270e862bbb272a3"
I20260925 03:03:21.357946 23948 raft_consensus.cc:3037] T c9f3dfa77c7845a49f80417befebb978 P e8496e8e22c447f9b270e862bbb272a3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:21.358240 24139 ts_tablet_manager.cc:1428] T 5ac7083552944834bb61700cfbe60faa P 1c37142ee9c043d7a5aca01030e1bbf5: Time spent starting tablet: real 0.008s user 0.005s sys 0.001s
I20260925 03:03:21.360248 24142 ts_tablet_manager.cc:1428] T 5055d207a0994afbbbc25a2c39455f87 P 5dae02dd94274f54ab9c52ab57a880b0: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260925 03:03:21.360582 24149 raft_consensus.cc:482] T 5055d207a0994afbbbc25a2c39455f87 P 5dae02dd94274f54ab9c52ab57a880b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:21.361156 24142 tablet_bootstrap.cc:492] T f58a83ec6f4149108eb0567cd7817ab5 P 5dae02dd94274f54ab9c52ab57a880b0: Bootstrap starting.
I20260925 03:03:21.361037 24149 raft_consensus.cc:504] T 5055d207a0994afbbbc25a2c39455f87 P 5dae02dd94274f54ab9c52ab57a880b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:21.364450 24139 tablet_bootstrap.cc:492] T be306de199ff4dfb976075219c02ea8e P 1c37142ee9c043d7a5aca01030e1bbf5: Bootstrap starting.
I20260925 03:03:21.368402 24142 tablet_bootstrap.cc:654] T f58a83ec6f4149108eb0567cd7817ab5 P 5dae02dd94274f54ab9c52ab57a880b0: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:21.368860 24152 raft_consensus.cc:504] T 7f1e503d876c45fa853edeaf3b836180 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:21.369189 24149 leader_election.cc:288] T 5055d207a0994afbbbc25a2c39455f87 P 5dae02dd94274f54ab9c52ab57a880b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e8496e8e22c447f9b270e862bbb272a3 (127.22.193.1:44975), 1c37142ee9c043d7a5aca01030e1bbf5 (127.22.193.2:40513)
I20260925 03:03:21.380548 24138 ts_tablet_manager.cc:1428] T 48c4fddba997473f8a501507277b8860 P e8496e8e22c447f9b270e862bbb272a3: Time spent starting tablet: real 0.078s user 0.007s sys 0.000s
I20260925 03:03:21.381707 24138 tablet_bootstrap.cc:492] T 78943162e8394d039f00afadd62893f3 P e8496e8e22c447f9b270e862bbb272a3: Bootstrap starting.
I20260925 03:03:21.382169 24152 leader_election.cc:288] T 7f1e503d876c45fa853edeaf3b836180 P 1c37142ee9c043d7a5aca01030e1bbf5 [CANDIDATE]: Term 1 election: Requested vote from peers e8496e8e22c447f9b270e862bbb272a3 (127.22.193.1:44975), 5dae02dd94274f54ab9c52ab57a880b0 (127.22.193.3:46383)
I20260925 03:03:21.383266 23949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f1e503d876c45fa853edeaf3b836180" candidate_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8496e8e22c447f9b270e862bbb272a3"
I20260925 03:03:21.384294 23948 raft_consensus.cc:2454] T c9f3dfa77c7845a49f80417befebb978 P e8496e8e22c447f9b270e862bbb272a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c37142ee9c043d7a5aca01030e1bbf5 in term 1.
I20260925 03:03:21.385216 23949 raft_consensus.cc:3037] T 7f1e503d876c45fa853edeaf3b836180 P e8496e8e22c447f9b270e862bbb272a3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:21.386080 24139 tablet_bootstrap.cc:654] T be306de199ff4dfb976075219c02ea8e P 1c37142ee9c043d7a5aca01030e1bbf5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:21.386750 23947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5055d207a0994afbbbc25a2c39455f87" candidate_uuid: "5dae02dd94274f54ab9c52ab57a880b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8496e8e22c447f9b270e862bbb272a3" is_pre_election: true
I20260925 03:03:21.392004 24138 tablet_bootstrap.cc:654] T 78943162e8394d039f00afadd62893f3 P e8496e8e22c447f9b270e862bbb272a3: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:21.397710 24142 tablet_bootstrap.cc:492] T f58a83ec6f4149108eb0567cd7817ab5 P 5dae02dd94274f54ab9c52ab57a880b0: No bootstrap required, opened a new log
I20260925 03:03:21.398308 24142 ts_tablet_manager.cc:1397] T f58a83ec6f4149108eb0567cd7817ab5 P 5dae02dd94274f54ab9c52ab57a880b0: Time spent bootstrapping tablet: real 0.037s user 0.011s sys 0.002s
I20260925 03:03:21.401238 24142 raft_consensus.cc:348] T f58a83ec6f4149108eb0567cd7817ab5 P 5dae02dd94274f54ab9c52ab57a880b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:21.402146 24142 raft_consensus.cc:374] T f58a83ec6f4149108eb0567cd7817ab5 P 5dae02dd94274f54ab9c52ab57a880b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:21.402529 24142 raft_consensus.cc:729] T f58a83ec6f4149108eb0567cd7817ab5 P 5dae02dd94274f54ab9c52ab57a880b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dae02dd94274f54ab9c52ab57a880b0, State: Initialized, Role: FOLLOWER
I20260925 03:03:21.403439 24142 consensus_queue.cc:260] T f58a83ec6f4149108eb0567cd7817ab5 P 5dae02dd94274f54ab9c52ab57a880b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:21.407826 24149 raft_consensus.cc:482] T 5ac7083552944834bb61700cfbe60faa P 5dae02dd94274f54ab9c52ab57a880b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:21.408334 24149 raft_consensus.cc:504] T 5ac7083552944834bb61700cfbe60faa P 5dae02dd94274f54ab9c52ab57a880b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:21.410360 24149 leader_election.cc:288] T 5ac7083552944834bb61700cfbe60faa P 5dae02dd94274f54ab9c52ab57a880b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e8496e8e22c447f9b270e862bbb272a3 (127.22.193.1:44975), 1c37142ee9c043d7a5aca01030e1bbf5 (127.22.193.2:40513)
I20260925 03:03:21.411480 23949 raft_consensus.cc:2454] T 7f1e503d876c45fa853edeaf3b836180 P e8496e8e22c447f9b270e862bbb272a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c37142ee9c043d7a5aca01030e1bbf5 in term 1.
I20260925 03:03:21.412559 24143 raft_consensus.cc:482] T 48c4fddba997473f8a501507277b8860 P e8496e8e22c447f9b270e862bbb272a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:21.413110 24143 raft_consensus.cc:504] T 48c4fddba997473f8a501507277b8860 P e8496e8e22c447f9b270e862bbb272a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:21.421238 24142 ts_tablet_manager.cc:1428] T f58a83ec6f4149108eb0567cd7817ab5 P 5dae02dd94274f54ab9c52ab57a880b0: Time spent starting tablet: real 0.023s user 0.006s sys 0.000s
I20260925 03:03:21.422075 24143 leader_election.cc:288] T 48c4fddba997473f8a501507277b8860 P e8496e8e22c447f9b270e862bbb272a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c37142ee9c043d7a5aca01030e1bbf5 (127.22.193.2:40513), 5dae02dd94274f54ab9c52ab57a880b0 (127.22.193.3:46383)
I20260925 03:03:21.426062 24142 tablet_bootstrap.cc:492] T 845902c057d54c5ab9ddc861f622125c P 5dae02dd94274f54ab9c52ab57a880b0: Bootstrap starting.
I20260925 03:03:21.432505 24139 tablet_bootstrap.cc:492] T be306de199ff4dfb976075219c02ea8e P 1c37142ee9c043d7a5aca01030e1bbf5: No bootstrap required, opened a new log
I20260925 03:03:21.433049 24139 ts_tablet_manager.cc:1397] T be306de199ff4dfb976075219c02ea8e P 1c37142ee9c043d7a5aca01030e1bbf5: Time spent bootstrapping tablet: real 0.069s user 0.011s sys 0.010s
I20260925 03:03:21.435775 24139 raft_consensus.cc:348] T be306de199ff4dfb976075219c02ea8e P 1c37142ee9c043d7a5aca01030e1bbf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:21.441725 24138 tablet_bootstrap.cc:492] T 78943162e8394d039f00afadd62893f3 P e8496e8e22c447f9b270e862bbb272a3: No bootstrap required, opened a new log
I20260925 03:03:21.445214 24138 ts_tablet_manager.cc:1397] T 78943162e8394d039f00afadd62893f3 P e8496e8e22c447f9b270e862bbb272a3: Time spent bootstrapping tablet: real 0.064s user 0.017s sys 0.003s
I20260925 03:03:21.436529 24139 raft_consensus.cc:374] T be306de199ff4dfb976075219c02ea8e P 1c37142ee9c043d7a5aca01030e1bbf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:21.447422 24139 raft_consensus.cc:729] T be306de199ff4dfb976075219c02ea8e P 1c37142ee9c043d7a5aca01030e1bbf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c37142ee9c043d7a5aca01030e1bbf5, State: Initialized, Role: FOLLOWER
I20260925 03:03:21.443969 24142 tablet_bootstrap.cc:654] T 845902c057d54c5ab9ddc861f622125c P 5dae02dd94274f54ab9c52ab57a880b0: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:21.448213 24139 consensus_queue.cc:260] T be306de199ff4dfb976075219c02ea8e P 1c37142ee9c043d7a5aca01030e1bbf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:21.451088 24138 raft_consensus.cc:348] T 78943162e8394d039f00afadd62893f3 P e8496e8e22c447f9b270e862bbb272a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:21.454394 24138 raft_consensus.cc:374] T 78943162e8394d039f00afadd62893f3 P e8496e8e22c447f9b270e862bbb272a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:21.454766 24138 raft_consensus.cc:729] T 78943162e8394d039f00afadd62893f3 P e8496e8e22c447f9b270e862bbb272a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8496e8e22c447f9b270e862bbb272a3, State: Initialized, Role: FOLLOWER
I20260925 03:03:21.455506 24138 consensus_queue.cc:260] T 78943162e8394d039f00afadd62893f3 P e8496e8e22c447f9b270e862bbb272a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:21.462348 24097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f1e503d876c45fa853edeaf3b836180" candidate_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dae02dd94274f54ab9c52ab57a880b0" is_pre_election: true
I20260925 03:03:21.463501 24138 ts_tablet_manager.cc:1428] T 78943162e8394d039f00afadd62893f3 P e8496e8e22c447f9b270e862bbb272a3: Time spent starting tablet: real 0.017s user 0.004s sys 0.001s
I20260925 03:03:21.464449 24138 tablet_bootstrap.cc:492] T 5ac7083552944834bb61700cfbe60faa P e8496e8e22c447f9b270e862bbb272a3: Bootstrap starting.
W20260925 03:03:21.451112 24062 leader_election.cc:341] T 5055d207a0994afbbbc25a2c39455f87 P 5dae02dd94274f54ab9c52ab57a880b0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e8496e8e22c447f9b270e862bbb272a3 (127.22.193.1:44975): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:21.458591 23988 leader_election.cc:302] T c9f3dfa77c7845a49f80417befebb978 P 1c37142ee9c043d7a5aca01030e1bbf5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c37142ee9c043d7a5aca01030e1bbf5, e8496e8e22c447f9b270e862bbb272a3; no voters:
I20260925 03:03:21.465662 24096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f1e503d876c45fa853edeaf3b836180" candidate_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dae02dd94274f54ab9c52ab57a880b0"
I20260925 03:03:21.466214 23988 leader_election.cc:302] T 7f1e503d876c45fa853edeaf3b836180 P 1c37142ee9c043d7a5aca01030e1bbf5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c37142ee9c043d7a5aca01030e1bbf5, e8496e8e22c447f9b270e862bbb272a3; no voters:
I20260925 03:03:21.466656 24152 raft_consensus.cc:2781] T c9f3dfa77c7845a49f80417befebb978 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:03:21.466697 23949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ac7083552944834bb61700cfbe60faa" candidate_uuid: "5dae02dd94274f54ab9c52ab57a880b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8496e8e22c447f9b270e862bbb272a3" is_pre_election: true
I20260925 03:03:21.467149 24152 raft_consensus.cc:686] T c9f3dfa77c7845a49f80417befebb978 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 1 LEADER]: Becoming Leader. State: Replica: 1c37142ee9c043d7a5aca01030e1bbf5, State: Running, Role: LEADER
I20260925 03:03:21.467931 24152 consensus_queue.cc:237] T c9f3dfa77c7845a49f80417befebb978 P 1c37142ee9c043d7a5aca01030e1bbf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:21.471164 24154 raft_consensus.cc:2781] T 7f1e503d876c45fa853edeaf3b836180 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:03:21.460865 24098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9f3dfa77c7845a49f80417befebb978" candidate_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dae02dd94274f54ab9c52ab57a880b0" is_pre_election: true
I20260925 03:03:21.464273 24097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9f3dfa77c7845a49f80417befebb978" candidate_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dae02dd94274f54ab9c52ab57a880b0"
I20260925 03:03:21.473519 24138 tablet_bootstrap.cc:654] T 5ac7083552944834bb61700cfbe60faa P e8496e8e22c447f9b270e862bbb272a3: Neither blocks nor log segments found. Creating new log.
W20260925 03:03:21.468324 24062 leader_election.cc:341] T 5ac7083552944834bb61700cfbe60faa P 5dae02dd94274f54ab9c52ab57a880b0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e8496e8e22c447f9b270e862bbb272a3 (127.22.193.1:44975): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:21.474828 24024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5055d207a0994afbbbc25a2c39455f87" candidate_uuid: "5dae02dd94274f54ab9c52ab57a880b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" is_pre_election: true
I20260925 03:03:21.476316 24023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ac7083552944834bb61700cfbe60faa" candidate_uuid: "5dae02dd94274f54ab9c52ab57a880b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" is_pre_election: true
I20260925 03:03:21.477052 24023 raft_consensus.cc:2454] T 5ac7083552944834bb61700cfbe60faa P 1c37142ee9c043d7a5aca01030e1bbf5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dae02dd94274f54ab9c52ab57a880b0 in term 0.
W20260925 03:03:21.477958 24062 leader_election.cc:341] T 5055d207a0994afbbbc25a2c39455f87 P 5dae02dd94274f54ab9c52ab57a880b0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c37142ee9c043d7a5aca01030e1bbf5 (127.22.193.2:40513): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:21.478392 24062 leader_election.cc:302] T 5055d207a0994afbbbc25a2c39455f87 P 5dae02dd94274f54ab9c52ab57a880b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5dae02dd94274f54ab9c52ab57a880b0; no voters: 1c37142ee9c043d7a5aca01030e1bbf5, e8496e8e22c447f9b270e862bbb272a3
I20260925 03:03:21.479328 24149 raft_consensus.cc:2726] T 5055d207a0994afbbbc25a2c39455f87 P 5dae02dd94274f54ab9c52ab57a880b0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 03:03:21.480630 24062 leader_election.cc:302] T 5ac7083552944834bb61700cfbe60faa P 5dae02dd94274f54ab9c52ab57a880b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c37142ee9c043d7a5aca01030e1bbf5, 5dae02dd94274f54ab9c52ab57a880b0; no voters: e8496e8e22c447f9b270e862bbb272a3
I20260925 03:03:21.481551 24149 raft_consensus.cc:2781] T 5ac7083552944834bb61700cfbe60faa P 5dae02dd94274f54ab9c52ab57a880b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 03:03:21.461732 24142 tablet_bootstrap.cc:492] T 845902c057d54c5ab9ddc861f622125c P 5dae02dd94274f54ab9c52ab57a880b0: No bootstrap required, opened a new log
I20260925 03:03:21.481886 24149 raft_consensus.cc:482] T 5ac7083552944834bb61700cfbe60faa P 5dae02dd94274f54ab9c52ab57a880b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:03:21.482187 24149 raft_consensus.cc:3037] T 5ac7083552944834bb61700cfbe60faa P 5dae02dd94274f54ab9c52ab57a880b0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:21.482712 24023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48c4fddba997473f8a501507277b8860" candidate_uuid: "e8496e8e22c447f9b270e862bbb272a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" is_pre_election: true
W20260925 03:03:21.484376 23913 leader_election.cc:341] T 48c4fddba997473f8a501507277b8860 P e8496e8e22c447f9b270e862bbb272a3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c37142ee9c043d7a5aca01030e1bbf5 (127.22.193.2:40513): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:21.487370 24097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48c4fddba997473f8a501507277b8860" candidate_uuid: "e8496e8e22c447f9b270e862bbb272a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dae02dd94274f54ab9c52ab57a880b0" is_pre_election: true
W20260925 03:03:21.488930 23915 leader_election.cc:341] T 48c4fddba997473f8a501507277b8860 P e8496e8e22c447f9b270e862bbb272a3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5dae02dd94274f54ab9c52ab57a880b0 (127.22.193.3:46383): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:21.489393 23915 leader_election.cc:302] T 48c4fddba997473f8a501507277b8860 P e8496e8e22c447f9b270e862bbb272a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e8496e8e22c447f9b270e862bbb272a3; no voters: 1c37142ee9c043d7a5aca01030e1bbf5, 5dae02dd94274f54ab9c52ab57a880b0
I20260925 03:03:21.489753 24149 raft_consensus.cc:504] T 5ac7083552944834bb61700cfbe60faa P 5dae02dd94274f54ab9c52ab57a880b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:21.482236 24142 ts_tablet_manager.cc:1397] T 845902c057d54c5ab9ddc861f622125c P 5dae02dd94274f54ab9c52ab57a880b0: Time spent bootstrapping tablet: real 0.056s user 0.011s sys 0.000s
I20260925 03:03:21.490513 24143 raft_consensus.cc:2726] T 48c4fddba997473f8a501507277b8860 P e8496e8e22c447f9b270e862bbb272a3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 03:03:21.490959 24154 raft_consensus.cc:686] T 7f1e503d876c45fa853edeaf3b836180 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 1 LEADER]: Becoming Leader. State: Replica: 1c37142ee9c043d7a5aca01030e1bbf5, State: Running, Role: LEADER
I20260925 03:03:21.493408 24154 consensus_queue.cc:237] T 7f1e503d876c45fa853edeaf3b836180 P 1c37142ee9c043d7a5aca01030e1bbf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:21.494578 24142 raft_consensus.cc:348] T 845902c057d54c5ab9ddc861f622125c P 5dae02dd94274f54ab9c52ab57a880b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:21.495338 24142 raft_consensus.cc:374] T 845902c057d54c5ab9ddc861f622125c P 5dae02dd94274f54ab9c52ab57a880b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:21.495677 24142 raft_consensus.cc:729] T 845902c057d54c5ab9ddc861f622125c P 5dae02dd94274f54ab9c52ab57a880b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dae02dd94274f54ab9c52ab57a880b0, State: Initialized, Role: FOLLOWER
I20260925 03:03:21.496778 24138 tablet_bootstrap.cc:492] T 5ac7083552944834bb61700cfbe60faa P e8496e8e22c447f9b270e862bbb272a3: No bootstrap required, opened a new log
I20260925 03:03:21.496634 23949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ac7083552944834bb61700cfbe60faa" candidate_uuid: "5dae02dd94274f54ab9c52ab57a880b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8496e8e22c447f9b270e862bbb272a3"
I20260925 03:03:21.497258 24138 ts_tablet_manager.cc:1397] T 5ac7083552944834bb61700cfbe60faa P e8496e8e22c447f9b270e862bbb272a3: Time spent bootstrapping tablet: real 0.033s user 0.012s sys 0.017s
W20260925 03:03:21.497634 23990 leader_election.cc:341] T 7f1e503d876c45fa853edeaf3b836180 P 1c37142ee9c043d7a5aca01030e1bbf5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5dae02dd94274f54ab9c52ab57a880b0 (127.22.193.3:46383): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:21.497848 24023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ac7083552944834bb61700cfbe60faa" candidate_uuid: "5dae02dd94274f54ab9c52ab57a880b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c37142ee9c043d7a5aca01030e1bbf5"
I20260925 03:03:21.497821 24142 consensus_queue.cc:260] T 845902c057d54c5ab9ddc861f622125c P 5dae02dd94274f54ab9c52ab57a880b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:21.498606 24023 raft_consensus.cc:3037] T 5ac7083552944834bb61700cfbe60faa P 1c37142ee9c043d7a5aca01030e1bbf5 [term 0 FOLLOWER]: Advancing to term 1
W20260925 03:03:21.498729 23990 leader_election.cc:341] T 7f1e503d876c45fa853edeaf3b836180 P 1c37142ee9c043d7a5aca01030e1bbf5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5dae02dd94274f54ab9c52ab57a880b0 (127.22.193.3:46383): Illegal state: must be running to vote when last-logged opid is not known
W20260925 03:03:21.499212 24062 leader_election.cc:341] T 5ac7083552944834bb61700cfbe60faa P 5dae02dd94274f54ab9c52ab57a880b0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e8496e8e22c447f9b270e862bbb272a3 (127.22.193.1:44975): Illegal state: must be running to vote when last-logged opid is not known
W20260925 03:03:21.499677 23990 leader_election.cc:341] T c9f3dfa77c7845a49f80417befebb978 P 1c37142ee9c043d7a5aca01030e1bbf5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5dae02dd94274f54ab9c52ab57a880b0 (127.22.193.3:46383): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:21.457459 24139 ts_tablet_manager.cc:1428] T be306de199ff4dfb976075219c02ea8e P 1c37142ee9c043d7a5aca01030e1bbf5: Time spent starting tablet: real 0.024s user 0.007s sys 0.000s
I20260925 03:03:21.499948 24149 leader_election.cc:288] T 5ac7083552944834bb61700cfbe60faa P 5dae02dd94274f54ab9c52ab57a880b0 [CANDIDATE]: Term 1 election: Requested vote from peers e8496e8e22c447f9b270e862bbb272a3 (127.22.193.1:44975), 1c37142ee9c043d7a5aca01030e1bbf5 (127.22.193.2:40513)
I20260925 03:03:21.499825 24138 raft_consensus.cc:348] T 5ac7083552944834bb61700cfbe60faa P e8496e8e22c447f9b270e862bbb272a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:21.500569 24138 raft_consensus.cc:374] T 5ac7083552944834bb61700cfbe60faa P e8496e8e22c447f9b270e862bbb272a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:21.500910 24138 raft_consensus.cc:729] T 5ac7083552944834bb61700cfbe60faa P e8496e8e22c447f9b270e862bbb272a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8496e8e22c447f9b270e862bbb272a3, State: Initialized, Role: FOLLOWER
W20260925 03:03:21.501552 23990 leader_election.cc:341] T c9f3dfa77c7845a49f80417befebb978 P 1c37142ee9c043d7a5aca01030e1bbf5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5dae02dd94274f54ab9c52ab57a880b0 (127.22.193.3:46383): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:21.501644 24138 consensus_queue.cc:260] T 5ac7083552944834bb61700cfbe60faa P e8496e8e22c447f9b270e862bbb272a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:21.500875 24139 tablet_bootstrap.cc:492] T 5055d207a0994afbbbc25a2c39455f87 P 1c37142ee9c043d7a5aca01030e1bbf5: Bootstrap starting.
I20260925 03:03:21.502741 24142 ts_tablet_manager.cc:1428] T 845902c057d54c5ab9ddc861f622125c P 5dae02dd94274f54ab9c52ab57a880b0: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
I20260925 03:03:21.503717 24142 tablet_bootstrap.cc:492] T 400a066077234276876ac579e8353ad2 P 5dae02dd94274f54ab9c52ab57a880b0: Bootstrap starting.
I20260925 03:03:21.503852 24138 ts_tablet_manager.cc:1428] T 5ac7083552944834bb61700cfbe60faa P e8496e8e22c447f9b270e862bbb272a3: Time spent starting tablet: real 0.006s user 0.001s sys 0.003s
I20260925 03:03:21.508728 24138 tablet_bootstrap.cc:492] T 845902c057d54c5ab9ddc861f622125c P e8496e8e22c447f9b270e862bbb272a3: Bootstrap starting.
I20260925 03:03:21.509883 24139 tablet_bootstrap.cc:654] T 5055d207a0994afbbbc25a2c39455f87 P 1c37142ee9c043d7a5aca01030e1bbf5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:21.509886 24142 tablet_bootstrap.cc:654] T 400a066077234276876ac579e8353ad2 P 5dae02dd94274f54ab9c52ab57a880b0: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:21.515321 24138 tablet_bootstrap.cc:654] T 845902c057d54c5ab9ddc861f622125c P e8496e8e22c447f9b270e862bbb272a3: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:21.532775 24023 raft_consensus.cc:2454] T 5ac7083552944834bb61700cfbe60faa P 1c37142ee9c043d7a5aca01030e1bbf5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dae02dd94274f54ab9c52ab57a880b0 in term 1.
I20260925 03:03:21.536509 24062 leader_election.cc:302] T 5ac7083552944834bb61700cfbe60faa P 5dae02dd94274f54ab9c52ab57a880b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c37142ee9c043d7a5aca01030e1bbf5, 5dae02dd94274f54ab9c52ab57a880b0; no voters: e8496e8e22c447f9b270e862bbb272a3
I20260925 03:03:21.537930 24149 raft_consensus.cc:2781] T 5ac7083552944834bb61700cfbe60faa P 5dae02dd94274f54ab9c52ab57a880b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:03:21.539490 24139 tablet_bootstrap.cc:492] T 5055d207a0994afbbbc25a2c39455f87 P 1c37142ee9c043d7a5aca01030e1bbf5: No bootstrap required, opened a new log
I20260925 03:03:21.539974 24139 ts_tablet_manager.cc:1397] T 5055d207a0994afbbbc25a2c39455f87 P 1c37142ee9c043d7a5aca01030e1bbf5: Time spent bootstrapping tablet: real 0.039s user 0.017s sys 0.012s
I20260925 03:03:21.541322 24142 tablet_bootstrap.cc:492] T 400a066077234276876ac579e8353ad2 P 5dae02dd94274f54ab9c52ab57a880b0: No bootstrap required, opened a new log
I20260925 03:03:21.541847 24142 ts_tablet_manager.cc:1397] T 400a066077234276876ac579e8353ad2 P 5dae02dd94274f54ab9c52ab57a880b0: Time spent bootstrapping tablet: real 0.038s user 0.013s sys 0.020s
I20260925 03:03:21.540977 23845 catalog_manager.cc:5495] T c9f3dfa77c7845a49f80417befebb978 P 1c37142ee9c043d7a5aca01030e1bbf5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c37142ee9c043d7a5aca01030e1bbf5 (127.22.193.2). New cstate: current_term: 1 leader_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } health_report { overall_health: UNKNOWN } } }
I20260925 03:03:21.542768 24139 raft_consensus.cc:348] T 5055d207a0994afbbbc25a2c39455f87 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:21.543912 24139 raft_consensus.cc:374] T 5055d207a0994afbbbc25a2c39455f87 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:21.543282 23845 catalog_manager.cc:5495] T 7f1e503d876c45fa853edeaf3b836180 P 1c37142ee9c043d7a5aca01030e1bbf5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c37142ee9c043d7a5aca01030e1bbf5 (127.22.193.2). New cstate: current_term: 1 leader_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } health_report { overall_health: UNKNOWN } } }
I20260925 03:03:21.544313 24139 raft_consensus.cc:729] T 5055d207a0994afbbbc25a2c39455f87 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c37142ee9c043d7a5aca01030e1bbf5, State: Initialized, Role: FOLLOWER
I20260925 03:03:21.544807 24142 raft_consensus.cc:348] T 400a066077234276876ac579e8353ad2 P 5dae02dd94274f54ab9c52ab57a880b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:21.545719 24142 raft_consensus.cc:374] T 400a066077234276876ac579e8353ad2 P 5dae02dd94274f54ab9c52ab57a880b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:21.545230 24139 consensus_queue.cc:260] T 5055d207a0994afbbbc25a2c39455f87 P 1c37142ee9c043d7a5aca01030e1bbf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:21.546020 24142 raft_consensus.cc:729] T 400a066077234276876ac579e8353ad2 P 5dae02dd94274f54ab9c52ab57a880b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dae02dd94274f54ab9c52ab57a880b0, State: Initialized, Role: FOLLOWER
I20260925 03:03:21.547073 24142 consensus_queue.cc:260] T 400a066077234276876ac579e8353ad2 P 5dae02dd94274f54ab9c52ab57a880b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:21.548254 24138 tablet_bootstrap.cc:492] T 845902c057d54c5ab9ddc861f622125c P e8496e8e22c447f9b270e862bbb272a3: No bootstrap required, opened a new log
I20260925 03:03:21.549218 24138 ts_tablet_manager.cc:1397] T 845902c057d54c5ab9ddc861f622125c P e8496e8e22c447f9b270e862bbb272a3: Time spent bootstrapping tablet: real 0.041s user 0.020s sys 0.008s
I20260925 03:03:21.548591 24139 ts_tablet_manager.cc:1428] T 5055d207a0994afbbbc25a2c39455f87 P 1c37142ee9c043d7a5aca01030e1bbf5: Time spent starting tablet: real 0.008s user 0.003s sys 0.003s
I20260925 03:03:21.553839 24139 tablet_bootstrap.cc:492] T f58a83ec6f4149108eb0567cd7817ab5 P 1c37142ee9c043d7a5aca01030e1bbf5: Bootstrap starting.
W20260925 03:03:21.555716 23895 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260925 03:03:21.558444 24149 raft_consensus.cc:686] T 5ac7083552944834bb61700cfbe60faa P 5dae02dd94274f54ab9c52ab57a880b0 [term 1 LEADER]: Becoming Leader. State: Replica: 5dae02dd94274f54ab9c52ab57a880b0, State: Running, Role: LEADER
I20260925 03:03:21.559368 24142 ts_tablet_manager.cc:1428] T 400a066077234276876ac579e8353ad2 P 5dae02dd94274f54ab9c52ab57a880b0: Time spent starting tablet: real 0.017s user 0.012s sys 0.003s
I20260925 03:03:21.552276 24138 raft_consensus.cc:348] T 845902c057d54c5ab9ddc861f622125c P e8496e8e22c447f9b270e862bbb272a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:21.559434 24149 consensus_queue.cc:237] T 5ac7083552944834bb61700cfbe60faa P 5dae02dd94274f54ab9c52ab57a880b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:21.560184 24138 raft_consensus.cc:374] T 845902c057d54c5ab9ddc861f622125c P e8496e8e22c447f9b270e862bbb272a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:21.560485 24138 raft_consensus.cc:729] T 845902c057d54c5ab9ddc861f622125c P e8496e8e22c447f9b270e862bbb272a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8496e8e22c447f9b270e862bbb272a3, State: Initialized, Role: FOLLOWER
W20260925 03:03:21.561079 24050 tablet.cc:2369] T 5055d207a0994afbbbc25a2c39455f87 P 1c37142ee9c043d7a5aca01030e1bbf5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 03:03:21.563508 24139 tablet_bootstrap.cc:654] T f58a83ec6f4149108eb0567cd7817ab5 P 1c37142ee9c043d7a5aca01030e1bbf5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:21.561127 24138 consensus_queue.cc:260] T 845902c057d54c5ab9ddc861f622125c P e8496e8e22c447f9b270e862bbb272a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:21.567432 24138 ts_tablet_manager.cc:1428] T 845902c057d54c5ab9ddc861f622125c P e8496e8e22c447f9b270e862bbb272a3: Time spent starting tablet: real 0.018s user 0.006s sys 0.000s
I20260925 03:03:21.568444 24138 tablet_bootstrap.cc:492] T be306de199ff4dfb976075219c02ea8e P e8496e8e22c447f9b270e862bbb272a3: Bootstrap starting.
I20260925 03:03:21.569592 24142 tablet_bootstrap.cc:492] T 7f1e503d876c45fa853edeaf3b836180 P 5dae02dd94274f54ab9c52ab57a880b0: Bootstrap starting.
I20260925 03:03:21.572510 24139 tablet_bootstrap.cc:492] T f58a83ec6f4149108eb0567cd7817ab5 P 1c37142ee9c043d7a5aca01030e1bbf5: No bootstrap required, opened a new log
I20260925 03:03:21.573062 24139 ts_tablet_manager.cc:1397] T f58a83ec6f4149108eb0567cd7817ab5 P 1c37142ee9c043d7a5aca01030e1bbf5: Time spent bootstrapping tablet: real 0.019s user 0.010s sys 0.000s
I20260925 03:03:21.575739 24139 raft_consensus.cc:348] T f58a83ec6f4149108eb0567cd7817ab5 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:21.576524 24139 raft_consensus.cc:374] T f58a83ec6f4149108eb0567cd7817ab5 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:21.576851 24139 raft_consensus.cc:729] T f58a83ec6f4149108eb0567cd7817ab5 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c37142ee9c043d7a5aca01030e1bbf5, State: Initialized, Role: FOLLOWER
I20260925 03:03:21.577711 24139 consensus_queue.cc:260] T f58a83ec6f4149108eb0567cd7817ab5 P 1c37142ee9c043d7a5aca01030e1bbf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:21.579567 24142 tablet_bootstrap.cc:654] T 7f1e503d876c45fa853edeaf3b836180 P 5dae02dd94274f54ab9c52ab57a880b0: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:21.582202 24138 tablet_bootstrap.cc:654] T be306de199ff4dfb976075219c02ea8e P e8496e8e22c447f9b270e862bbb272a3: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:21.580291 24139 ts_tablet_manager.cc:1428] T f58a83ec6f4149108eb0567cd7817ab5 P 1c37142ee9c043d7a5aca01030e1bbf5: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260925 03:03:21.584100 24139 tablet_bootstrap.cc:492] T 845902c057d54c5ab9ddc861f622125c P 1c37142ee9c043d7a5aca01030e1bbf5: Bootstrap starting.
I20260925 03:03:21.586591 24142 tablet_bootstrap.cc:492] T 7f1e503d876c45fa853edeaf3b836180 P 5dae02dd94274f54ab9c52ab57a880b0: No bootstrap required, opened a new log
I20260925 03:03:21.587132 24142 ts_tablet_manager.cc:1397] T 7f1e503d876c45fa853edeaf3b836180 P 5dae02dd94274f54ab9c52ab57a880b0: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.007s
I20260925 03:03:21.590075 24139 tablet_bootstrap.cc:654] T 845902c057d54c5ab9ddc861f622125c P 1c37142ee9c043d7a5aca01030e1bbf5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:21.590080 24142 raft_consensus.cc:348] T 7f1e503d876c45fa853edeaf3b836180 P 5dae02dd94274f54ab9c52ab57a880b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:21.591055 24142 raft_consensus.cc:374] T 7f1e503d876c45fa853edeaf3b836180 P 5dae02dd94274f54ab9c52ab57a880b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:21.591395 24138 tablet_bootstrap.cc:492] T be306de199ff4dfb976075219c02ea8e P e8496e8e22c447f9b270e862bbb272a3: No bootstrap required, opened a new log
I20260925 03:03:21.592001 24138 ts_tablet_manager.cc:1397] T be306de199ff4dfb976075219c02ea8e P e8496e8e22c447f9b270e862bbb272a3: Time spent bootstrapping tablet: real 0.024s user 0.014s sys 0.000s
I20260925 03:03:21.593032 24143 raft_consensus.cc:482] T 5ac7083552944834bb61700cfbe60faa P e8496e8e22c447f9b270e862bbb272a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:21.593463 24143 raft_consensus.cc:504] T 5ac7083552944834bb61700cfbe60faa P e8496e8e22c447f9b270e862bbb272a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:21.595383 24143 leader_election.cc:288] T 5ac7083552944834bb61700cfbe60faa P e8496e8e22c447f9b270e862bbb272a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c37142ee9c043d7a5aca01030e1bbf5 (127.22.193.2:40513), 5dae02dd94274f54ab9c52ab57a880b0 (127.22.193.3:46383)
I20260925 03:03:21.591348 24142 raft_consensus.cc:729] T 7f1e503d876c45fa853edeaf3b836180 P 5dae02dd94274f54ab9c52ab57a880b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dae02dd94274f54ab9c52ab57a880b0, State: Initialized, Role: FOLLOWER
I20260925 03:03:21.596331 24142 consensus_queue.cc:260] T 7f1e503d876c45fa853edeaf3b836180 P 5dae02dd94274f54ab9c52ab57a880b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:21.597461 24023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ac7083552944834bb61700cfbe60faa" candidate_uuid: "e8496e8e22c447f9b270e862bbb272a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" is_pre_election: true
I20260925 03:03:21.598533 24023 raft_consensus.cc:2379] T 5ac7083552944834bb61700cfbe60faa P 1c37142ee9c043d7a5aca01030e1bbf5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e8496e8e22c447f9b270e862bbb272a3 in current term 1: Already voted for candidate 5dae02dd94274f54ab9c52ab57a880b0 in this term.
I20260925 03:03:21.597461 24097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ac7083552944834bb61700cfbe60faa" candidate_uuid: "e8496e8e22c447f9b270e862bbb272a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dae02dd94274f54ab9c52ab57a880b0" is_pre_election: true
I20260925 03:03:21.600188 24142 ts_tablet_manager.cc:1428] T 7f1e503d876c45fa853edeaf3b836180 P 5dae02dd94274f54ab9c52ab57a880b0: Time spent starting tablet: real 0.013s user 0.005s sys 0.004s
I20260925 03:03:21.599944 23847 catalog_manager.cc:5495] T 5ac7083552944834bb61700cfbe60faa P 5dae02dd94274f54ab9c52ab57a880b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5dae02dd94274f54ab9c52ab57a880b0 (127.22.193.3). New cstate: current_term: 1 leader_uuid: "5dae02dd94274f54ab9c52ab57a880b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } health_report { overall_health: HEALTHY } } }
I20260925 03:03:21.600721 23915 leader_election.cc:302] T 5ac7083552944834bb61700cfbe60faa P e8496e8e22c447f9b270e862bbb272a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e8496e8e22c447f9b270e862bbb272a3; no voters: 1c37142ee9c043d7a5aca01030e1bbf5, 5dae02dd94274f54ab9c52ab57a880b0
I20260925 03:03:21.601351 24142 tablet_bootstrap.cc:492] T 48c4fddba997473f8a501507277b8860 P 5dae02dd94274f54ab9c52ab57a880b0: Bootstrap starting.
I20260925 03:03:21.601794 24143 raft_consensus.cc:3037] T 5ac7083552944834bb61700cfbe60faa P e8496e8e22c447f9b270e862bbb272a3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:21.597362 24138 raft_consensus.cc:348] T be306de199ff4dfb976075219c02ea8e P e8496e8e22c447f9b270e862bbb272a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:21.603060 24138 raft_consensus.cc:374] T be306de199ff4dfb976075219c02ea8e P e8496e8e22c447f9b270e862bbb272a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:21.603349 24138 raft_consensus.cc:729] T be306de199ff4dfb976075219c02ea8e P e8496e8e22c447f9b270e862bbb272a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8496e8e22c447f9b270e862bbb272a3, State: Initialized, Role: FOLLOWER
I20260925 03:03:21.604022 24138 consensus_queue.cc:260] T be306de199ff4dfb976075219c02ea8e P e8496e8e22c447f9b270e862bbb272a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:21.608140 24142 tablet_bootstrap.cc:654] T 48c4fddba997473f8a501507277b8860 P 5dae02dd94274f54ab9c52ab57a880b0: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:21.612142 24139 tablet_bootstrap.cc:492] T 845902c057d54c5ab9ddc861f622125c P 1c37142ee9c043d7a5aca01030e1bbf5: No bootstrap required, opened a new log
I20260925 03:03:21.612564 24143 raft_consensus.cc:2726] T 5ac7083552944834bb61700cfbe60faa P e8496e8e22c447f9b270e862bbb272a3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 03:03:21.612705 24139 ts_tablet_manager.cc:1397] T 845902c057d54c5ab9ddc861f622125c P 1c37142ee9c043d7a5aca01030e1bbf5: Time spent bootstrapping tablet: real 0.029s user 0.008s sys 0.008s
I20260925 03:03:21.613229 24138 ts_tablet_manager.cc:1428] T be306de199ff4dfb976075219c02ea8e P e8496e8e22c447f9b270e862bbb272a3: Time spent starting tablet: real 0.021s user 0.004s sys 0.008s
I20260925 03:03:21.614084 24138 tablet_bootstrap.cc:492] T f58a83ec6f4149108eb0567cd7817ab5 P e8496e8e22c447f9b270e862bbb272a3: Bootstrap starting.
I20260925 03:03:21.615572 24139 raft_consensus.cc:348] T 845902c057d54c5ab9ddc861f622125c P 1c37142ee9c043d7a5aca01030e1bbf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:21.616297 24139 raft_consensus.cc:374] T 845902c057d54c5ab9ddc861f622125c P 1c37142ee9c043d7a5aca01030e1bbf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:21.616587 24139 raft_consensus.cc:729] T 845902c057d54c5ab9ddc861f622125c P 1c37142ee9c043d7a5aca01030e1bbf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c37142ee9c043d7a5aca01030e1bbf5, State: Initialized, Role: FOLLOWER
I20260925 03:03:21.617353 24139 consensus_queue.cc:260] T 845902c057d54c5ab9ddc861f622125c P 1c37142ee9c043d7a5aca01030e1bbf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:21.619712 24138 tablet_bootstrap.cc:654] T f58a83ec6f4149108eb0567cd7817ab5 P e8496e8e22c447f9b270e862bbb272a3: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:21.623452 24154 raft_consensus.cc:482] T be306de199ff4dfb976075219c02ea8e P 1c37142ee9c043d7a5aca01030e1bbf5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:21.623927 24154 raft_consensus.cc:504] T be306de199ff4dfb976075219c02ea8e P 1c37142ee9c043d7a5aca01030e1bbf5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:21.625859 24154 leader_election.cc:288] T be306de199ff4dfb976075219c02ea8e P 1c37142ee9c043d7a5aca01030e1bbf5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e8496e8e22c447f9b270e862bbb272a3 (127.22.193.1:44975), 5dae02dd94274f54ab9c52ab57a880b0 (127.22.193.3:46383)
I20260925 03:03:21.627373 23949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be306de199ff4dfb976075219c02ea8e" candidate_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8496e8e22c447f9b270e862bbb272a3" is_pre_election: true
I20260925 03:03:21.627462 24097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be306de199ff4dfb976075219c02ea8e" candidate_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dae02dd94274f54ab9c52ab57a880b0" is_pre_election: true
I20260925 03:03:21.628113 23949 raft_consensus.cc:2454] T be306de199ff4dfb976075219c02ea8e P e8496e8e22c447f9b270e862bbb272a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c37142ee9c043d7a5aca01030e1bbf5 in term 0.
W20260925 03:03:21.629041 23990 leader_election.cc:341] T be306de199ff4dfb976075219c02ea8e P 1c37142ee9c043d7a5aca01030e1bbf5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5dae02dd94274f54ab9c52ab57a880b0 (127.22.193.3:46383): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:21.629678 23988 leader_election.cc:302] T be306de199ff4dfb976075219c02ea8e P 1c37142ee9c043d7a5aca01030e1bbf5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c37142ee9c043d7a5aca01030e1bbf5, e8496e8e22c447f9b270e862bbb272a3; no voters: 5dae02dd94274f54ab9c52ab57a880b0
I20260925 03:03:21.630919 24154 raft_consensus.cc:2781] T be306de199ff4dfb976075219c02ea8e P 1c37142ee9c043d7a5aca01030e1bbf5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 03:03:21.631326 24154 raft_consensus.cc:482] T be306de199ff4dfb976075219c02ea8e P 1c37142ee9c043d7a5aca01030e1bbf5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:03:21.632462 24142 tablet_bootstrap.cc:492] T 48c4fddba997473f8a501507277b8860 P 5dae02dd94274f54ab9c52ab57a880b0: No bootstrap required, opened a new log
I20260925 03:03:21.632653 24154 raft_consensus.cc:3037] T be306de199ff4dfb976075219c02ea8e P 1c37142ee9c043d7a5aca01030e1bbf5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:21.632980 24142 ts_tablet_manager.cc:1397] T 48c4fddba997473f8a501507277b8860 P 5dae02dd94274f54ab9c52ab57a880b0: Time spent bootstrapping tablet: real 0.032s user 0.012s sys 0.002s
I20260925 03:03:21.633152 24139 ts_tablet_manager.cc:1428] T 845902c057d54c5ab9ddc861f622125c P 1c37142ee9c043d7a5aca01030e1bbf5: Time spent starting tablet: real 0.020s user 0.006s sys 0.000s
I20260925 03:03:21.634277 24139 tablet_bootstrap.cc:492] T 400a066077234276876ac579e8353ad2 P 1c37142ee9c043d7a5aca01030e1bbf5: Bootstrap starting.
I20260925 03:03:21.635617 24142 raft_consensus.cc:348] T 48c4fddba997473f8a501507277b8860 P 5dae02dd94274f54ab9c52ab57a880b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:21.636371 24142 raft_consensus.cc:374] T 48c4fddba997473f8a501507277b8860 P 5dae02dd94274f54ab9c52ab57a880b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:21.636657 24142 raft_consensus.cc:729] T 48c4fddba997473f8a501507277b8860 P 5dae02dd94274f54ab9c52ab57a880b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dae02dd94274f54ab9c52ab57a880b0, State: Initialized, Role: FOLLOWER
I20260925 03:03:21.637529 24142 consensus_queue.cc:260] T 48c4fddba997473f8a501507277b8860 P 5dae02dd94274f54ab9c52ab57a880b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:21.642567 24139 tablet_bootstrap.cc:654] T 400a066077234276876ac579e8353ad2 P 1c37142ee9c043d7a5aca01030e1bbf5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:21.647007 24154 raft_consensus.cc:504] T be306de199ff4dfb976075219c02ea8e P 1c37142ee9c043d7a5aca01030e1bbf5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:21.650085 23949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be306de199ff4dfb976075219c02ea8e" candidate_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8496e8e22c447f9b270e862bbb272a3"
I20260925 03:03:21.650689 23949 raft_consensus.cc:3037] T be306de199ff4dfb976075219c02ea8e P e8496e8e22c447f9b270e862bbb272a3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:21.651129 24142 ts_tablet_manager.cc:1428] T 48c4fddba997473f8a501507277b8860 P 5dae02dd94274f54ab9c52ab57a880b0: Time spent starting tablet: real 0.018s user 0.005s sys 0.000s
I20260925 03:03:21.652196 24142 tablet_bootstrap.cc:492] T c9f3dfa77c7845a49f80417befebb978 P 5dae02dd94274f54ab9c52ab57a880b0: Bootstrap starting.
I20260925 03:03:21.655202 24143 raft_consensus.cc:482] T 78943162e8394d039f00afadd62893f3 P e8496e8e22c447f9b270e862bbb272a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:21.655715 24143 raft_consensus.cc:504] T 78943162e8394d039f00afadd62893f3 P e8496e8e22c447f9b270e862bbb272a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:21.657935 24143 leader_election.cc:288] T 78943162e8394d039f00afadd62893f3 P e8496e8e22c447f9b270e862bbb272a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c37142ee9c043d7a5aca01030e1bbf5 (127.22.193.2:40513), 5dae02dd94274f54ab9c52ab57a880b0 (127.22.193.3:46383)
I20260925 03:03:21.658862 24142 tablet_bootstrap.cc:654] T c9f3dfa77c7845a49f80417befebb978 P 5dae02dd94274f54ab9c52ab57a880b0: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:21.661013 24097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78943162e8394d039f00afadd62893f3" candidate_uuid: "e8496e8e22c447f9b270e862bbb272a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dae02dd94274f54ab9c52ab57a880b0" is_pre_election: true
W20260925 03:03:21.664739 23915 leader_election.cc:341] T 78943162e8394d039f00afadd62893f3 P e8496e8e22c447f9b270e862bbb272a3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5dae02dd94274f54ab9c52ab57a880b0 (127.22.193.3:46383): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:21.665756 24023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78943162e8394d039f00afadd62893f3" candidate_uuid: "e8496e8e22c447f9b270e862bbb272a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" is_pre_election: true
W20260925 03:03:21.667351 23913 leader_election.cc:341] T 78943162e8394d039f00afadd62893f3 P e8496e8e22c447f9b270e862bbb272a3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c37142ee9c043d7a5aca01030e1bbf5 (127.22.193.2:40513): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:21.667799 23913 leader_election.cc:302] T 78943162e8394d039f00afadd62893f3 P e8496e8e22c447f9b270e862bbb272a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e8496e8e22c447f9b270e862bbb272a3; no voters: 1c37142ee9c043d7a5aca01030e1bbf5, 5dae02dd94274f54ab9c52ab57a880b0
I20260925 03:03:21.669032 24143 raft_consensus.cc:2726] T 78943162e8394d039f00afadd62893f3 P e8496e8e22c447f9b270e862bbb272a3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 03:03:21.670054 23949 raft_consensus.cc:2454] T be306de199ff4dfb976075219c02ea8e P e8496e8e22c447f9b270e862bbb272a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c37142ee9c043d7a5aca01030e1bbf5 in term 1.
I20260925 03:03:21.671389 23988 leader_election.cc:302] T be306de199ff4dfb976075219c02ea8e P 1c37142ee9c043d7a5aca01030e1bbf5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c37142ee9c043d7a5aca01030e1bbf5, e8496e8e22c447f9b270e862bbb272a3; no voters:
I20260925 03:03:21.672022 24138 tablet_bootstrap.cc:492] T f58a83ec6f4149108eb0567cd7817ab5 P e8496e8e22c447f9b270e862bbb272a3: No bootstrap required, opened a new log
I20260925 03:03:21.672384 24152 raft_consensus.cc:2781] T be306de199ff4dfb976075219c02ea8e P 1c37142ee9c043d7a5aca01030e1bbf5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:03:21.672549 24138 ts_tablet_manager.cc:1397] T f58a83ec6f4149108eb0567cd7817ab5 P e8496e8e22c447f9b270e862bbb272a3: Time spent bootstrapping tablet: real 0.059s user 0.027s sys 0.013s
I20260925 03:03:21.672987 24152 raft_consensus.cc:686] T be306de199ff4dfb976075219c02ea8e P 1c37142ee9c043d7a5aca01030e1bbf5 [term 1 LEADER]: Becoming Leader. State: Replica: 1c37142ee9c043d7a5aca01030e1bbf5, State: Running, Role: LEADER
I20260925 03:03:21.673909 24152 consensus_queue.cc:237] T be306de199ff4dfb976075219c02ea8e P 1c37142ee9c043d7a5aca01030e1bbf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:21.675758 24138 raft_consensus.cc:348] T f58a83ec6f4149108eb0567cd7817ab5 P e8496e8e22c447f9b270e862bbb272a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:21.676535 24138 raft_consensus.cc:374] T f58a83ec6f4149108eb0567cd7817ab5 P e8496e8e22c447f9b270e862bbb272a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:21.676823 24138 raft_consensus.cc:729] T f58a83ec6f4149108eb0567cd7817ab5 P e8496e8e22c447f9b270e862bbb272a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8496e8e22c447f9b270e862bbb272a3, State: Initialized, Role: FOLLOWER
I20260925 03:03:21.677590 24138 consensus_queue.cc:260] T f58a83ec6f4149108eb0567cd7817ab5 P e8496e8e22c447f9b270e862bbb272a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:21.681551 24097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be306de199ff4dfb976075219c02ea8e" candidate_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dae02dd94274f54ab9c52ab57a880b0"
W20260925 03:03:21.683350 23990 leader_election.cc:341] T be306de199ff4dfb976075219c02ea8e P 1c37142ee9c043d7a5aca01030e1bbf5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5dae02dd94274f54ab9c52ab57a880b0 (127.22.193.3:46383): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:21.684341 24149 raft_consensus.cc:482] T 845902c057d54c5ab9ddc861f622125c P 5dae02dd94274f54ab9c52ab57a880b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:21.684935 24149 raft_consensus.cc:504] T 845902c057d54c5ab9ddc861f622125c P 5dae02dd94274f54ab9c52ab57a880b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:21.687130 24149 leader_election.cc:288] T 845902c057d54c5ab9ddc861f622125c P 5dae02dd94274f54ab9c52ab57a880b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e8496e8e22c447f9b270e862bbb272a3 (127.22.193.1:44975), 1c37142ee9c043d7a5aca01030e1bbf5 (127.22.193.2:40513)
I20260925 03:03:21.687572 24138 ts_tablet_manager.cc:1428] T f58a83ec6f4149108eb0567cd7817ab5 P e8496e8e22c447f9b270e862bbb272a3: Time spent starting tablet: real 0.015s user 0.005s sys 0.000s
I20260925 03:03:21.690004 24023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "845902c057d54c5ab9ddc861f622125c" candidate_uuid: "5dae02dd94274f54ab9c52ab57a880b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" is_pre_election: true
I20260925 03:03:21.690790 24023 raft_consensus.cc:2454] T 845902c057d54c5ab9ddc861f622125c P 1c37142ee9c043d7a5aca01030e1bbf5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dae02dd94274f54ab9c52ab57a880b0 in term 0.
I20260925 03:03:21.692283 24062 leader_election.cc:302] T 845902c057d54c5ab9ddc861f622125c P 5dae02dd94274f54ab9c52ab57a880b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c37142ee9c043d7a5aca01030e1bbf5, 5dae02dd94274f54ab9c52ab57a880b0; no voters:
I20260925 03:03:21.693400 24149 raft_consensus.cc:2781] T 845902c057d54c5ab9ddc861f622125c P 5dae02dd94274f54ab9c52ab57a880b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 03:03:21.693821 24149 raft_consensus.cc:482] T 845902c057d54c5ab9ddc861f622125c P 5dae02dd94274f54ab9c52ab57a880b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:03:21.694237 24149 raft_consensus.cc:3037] T 845902c057d54c5ab9ddc861f622125c P 5dae02dd94274f54ab9c52ab57a880b0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:21.696158 24138 tablet_bootstrap.cc:492] T 400a066077234276876ac579e8353ad2 P e8496e8e22c447f9b270e862bbb272a3: Bootstrap starting.
I20260925 03:03:21.701149 23949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "845902c057d54c5ab9ddc861f622125c" candidate_uuid: "5dae02dd94274f54ab9c52ab57a880b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8496e8e22c447f9b270e862bbb272a3" is_pre_election: true
I20260925 03:03:21.702363 24138 tablet_bootstrap.cc:654] T 400a066077234276876ac579e8353ad2 P e8496e8e22c447f9b270e862bbb272a3: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:21.703003 24154 leader_election.cc:288] T be306de199ff4dfb976075219c02ea8e P 1c37142ee9c043d7a5aca01030e1bbf5 [CANDIDATE]: Term 1 election: Requested vote from peers e8496e8e22c447f9b270e862bbb272a3 (127.22.193.1:44975), 5dae02dd94274f54ab9c52ab57a880b0 (127.22.193.3:46383)
I20260925 03:03:21.702298 23949 raft_consensus.cc:2454] T 845902c057d54c5ab9ddc861f622125c P e8496e8e22c447f9b270e862bbb272a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dae02dd94274f54ab9c52ab57a880b0 in term 0.
I20260925 03:03:21.708329 24149 raft_consensus.cc:504] T 845902c057d54c5ab9ddc861f622125c P 5dae02dd94274f54ab9c52ab57a880b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:21.715224 24138 tablet_bootstrap.cc:492] T 400a066077234276876ac579e8353ad2 P e8496e8e22c447f9b270e862bbb272a3: No bootstrap required, opened a new log
I20260925 03:03:21.715758 24138 ts_tablet_manager.cc:1397] T 400a066077234276876ac579e8353ad2 P e8496e8e22c447f9b270e862bbb272a3: Time spent bootstrapping tablet: real 0.020s user 0.010s sys 0.005s
I20260925 03:03:21.718308 24138 raft_consensus.cc:348] T 400a066077234276876ac579e8353ad2 P e8496e8e22c447f9b270e862bbb272a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:21.719588 24138 raft_consensus.cc:374] T 400a066077234276876ac579e8353ad2 P e8496e8e22c447f9b270e862bbb272a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:21.719944 24138 raft_consensus.cc:729] T 400a066077234276876ac579e8353ad2 P e8496e8e22c447f9b270e862bbb272a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8496e8e22c447f9b270e862bbb272a3, State: Initialized, Role: FOLLOWER
I20260925 03:03:21.720803 24138 consensus_queue.cc:260] T 400a066077234276876ac579e8353ad2 P e8496e8e22c447f9b270e862bbb272a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:21.722179 23847 catalog_manager.cc:5495] T be306de199ff4dfb976075219c02ea8e P 1c37142ee9c043d7a5aca01030e1bbf5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c37142ee9c043d7a5aca01030e1bbf5 (127.22.193.2). New cstate: current_term: 1 leader_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } health_report { overall_health: UNKNOWN } } }
I20260925 03:03:21.724887 24172 raft_consensus.cc:482] T f58a83ec6f4149108eb0567cd7817ab5 P 5dae02dd94274f54ab9c52ab57a880b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:21.725562 24172 raft_consensus.cc:504] T f58a83ec6f4149108eb0567cd7817ab5 P 5dae02dd94274f54ab9c52ab57a880b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:21.727720 24172 leader_election.cc:288] T f58a83ec6f4149108eb0567cd7817ab5 P 5dae02dd94274f54ab9c52ab57a880b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c37142ee9c043d7a5aca01030e1bbf5 (127.22.193.2:40513), e8496e8e22c447f9b270e862bbb272a3 (127.22.193.1:44975)
I20260925 03:03:21.729768 24023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f58a83ec6f4149108eb0567cd7817ab5" candidate_uuid: "5dae02dd94274f54ab9c52ab57a880b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" is_pre_election: true
I20260925 03:03:21.730558 24023 raft_consensus.cc:2454] T f58a83ec6f4149108eb0567cd7817ab5 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dae02dd94274f54ab9c52ab57a880b0 in term 0.
I20260925 03:03:21.731187 24138 ts_tablet_manager.cc:1428] T 400a066077234276876ac579e8353ad2 P e8496e8e22c447f9b270e862bbb272a3: Time spent starting tablet: real 0.015s user 0.007s sys 0.000s
I20260925 03:03:21.732286 24138 tablet_bootstrap.cc:492] T 5055d207a0994afbbbc25a2c39455f87 P e8496e8e22c447f9b270e862bbb272a3: Bootstrap starting.
I20260925 03:03:21.735491 24062 leader_election.cc:302] T f58a83ec6f4149108eb0567cd7817ab5 P 5dae02dd94274f54ab9c52ab57a880b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c37142ee9c043d7a5aca01030e1bbf5, 5dae02dd94274f54ab9c52ab57a880b0; no voters:
I20260925 03:03:21.737174 23949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f58a83ec6f4149108eb0567cd7817ab5" candidate_uuid: "5dae02dd94274f54ab9c52ab57a880b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8496e8e22c447f9b270e862bbb272a3" is_pre_election: true
I20260925 03:03:21.737926 23949 raft_consensus.cc:2454] T f58a83ec6f4149108eb0567cd7817ab5 P e8496e8e22c447f9b270e862bbb272a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dae02dd94274f54ab9c52ab57a880b0 in term 0.
I20260925 03:03:21.739955 24172 raft_consensus.cc:2781] T f58a83ec6f4149108eb0567cd7817ab5 P 5dae02dd94274f54ab9c52ab57a880b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 03:03:21.740388 24172 raft_consensus.cc:482] T f58a83ec6f4149108eb0567cd7817ab5 P 5dae02dd94274f54ab9c52ab57a880b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:03:21.740743 24172 raft_consensus.cc:3037] T f58a83ec6f4149108eb0567cd7817ab5 P 5dae02dd94274f54ab9c52ab57a880b0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:21.744766 24138 tablet_bootstrap.cc:654] T 5055d207a0994afbbbc25a2c39455f87 P e8496e8e22c447f9b270e862bbb272a3: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:21.746628 24149 leader_election.cc:288] T 845902c057d54c5ab9ddc861f622125c P 5dae02dd94274f54ab9c52ab57a880b0 [CANDIDATE]: Term 1 election: Requested vote from peers e8496e8e22c447f9b270e862bbb272a3 (127.22.193.1:44975), 1c37142ee9c043d7a5aca01030e1bbf5 (127.22.193.2:40513)
I20260925 03:03:21.749258 24023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "845902c057d54c5ab9ddc861f622125c" candidate_uuid: "5dae02dd94274f54ab9c52ab57a880b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c37142ee9c043d7a5aca01030e1bbf5"
I20260925 03:03:21.749959 24023 raft_consensus.cc:3037] T 845902c057d54c5ab9ddc861f622125c P 1c37142ee9c043d7a5aca01030e1bbf5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:21.757251 23949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "845902c057d54c5ab9ddc861f622125c" candidate_uuid: "5dae02dd94274f54ab9c52ab57a880b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8496e8e22c447f9b270e862bbb272a3"
I20260925 03:03:21.757992 23949 raft_consensus.cc:3037] T 845902c057d54c5ab9ddc861f622125c P e8496e8e22c447f9b270e862bbb272a3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:21.775094 24023 raft_consensus.cc:2454] T 845902c057d54c5ab9ddc861f622125c P 1c37142ee9c043d7a5aca01030e1bbf5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dae02dd94274f54ab9c52ab57a880b0 in term 1.
I20260925 03:03:21.779384 24062 leader_election.cc:302] T 845902c057d54c5ab9ddc861f622125c P 5dae02dd94274f54ab9c52ab57a880b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c37142ee9c043d7a5aca01030e1bbf5, 5dae02dd94274f54ab9c52ab57a880b0; no voters:
I20260925 03:03:21.780727 24149 raft_consensus.cc:2781] T 845902c057d54c5ab9ddc861f622125c P 5dae02dd94274f54ab9c52ab57a880b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:03:21.781246 24149 raft_consensus.cc:686] T 845902c057d54c5ab9ddc861f622125c P 5dae02dd94274f54ab9c52ab57a880b0 [term 1 LEADER]: Becoming Leader. State: Replica: 5dae02dd94274f54ab9c52ab57a880b0, State: Running, Role: LEADER
I20260925 03:03:21.782068 24149 consensus_queue.cc:237] T 845902c057d54c5ab9ddc861f622125c P 5dae02dd94274f54ab9c52ab57a880b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:21.783670 24138 tablet_bootstrap.cc:492] T 5055d207a0994afbbbc25a2c39455f87 P e8496e8e22c447f9b270e862bbb272a3: No bootstrap required, opened a new log
I20260925 03:03:21.784063 24139 tablet_bootstrap.cc:492] T 400a066077234276876ac579e8353ad2 P 1c37142ee9c043d7a5aca01030e1bbf5: No bootstrap required, opened a new log
I20260925 03:03:21.784189 24138 ts_tablet_manager.cc:1397] T 5055d207a0994afbbbc25a2c39455f87 P e8496e8e22c447f9b270e862bbb272a3: Time spent bootstrapping tablet: real 0.052s user 0.020s sys 0.023s
I20260925 03:03:21.784543 24139 ts_tablet_manager.cc:1397] T 400a066077234276876ac579e8353ad2 P 1c37142ee9c043d7a5aca01030e1bbf5: Time spent bootstrapping tablet: real 0.150s user 0.060s sys 0.077s
I20260925 03:03:21.784837 24172 raft_consensus.cc:504] T f58a83ec6f4149108eb0567cd7817ab5 P 5dae02dd94274f54ab9c52ab57a880b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:21.786846 24138 raft_consensus.cc:348] T 5055d207a0994afbbbc25a2c39455f87 P e8496e8e22c447f9b270e862bbb272a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:21.786984 24139 raft_consensus.cc:348] T 400a066077234276876ac579e8353ad2 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:21.787030 23949 raft_consensus.cc:2454] T 845902c057d54c5ab9ddc861f622125c P e8496e8e22c447f9b270e862bbb272a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dae02dd94274f54ab9c52ab57a880b0 in term 1.
I20260925 03:03:21.787662 24139 raft_consensus.cc:374] T 400a066077234276876ac579e8353ad2 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:21.787662 24138 raft_consensus.cc:374] T 5055d207a0994afbbbc25a2c39455f87 P e8496e8e22c447f9b270e862bbb272a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:21.788084 24139 raft_consensus.cc:729] T 400a066077234276876ac579e8353ad2 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c37142ee9c043d7a5aca01030e1bbf5, State: Initialized, Role: FOLLOWER
I20260925 03:03:21.788201 24138 raft_consensus.cc:729] T 5055d207a0994afbbbc25a2c39455f87 P e8496e8e22c447f9b270e862bbb272a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8496e8e22c447f9b270e862bbb272a3, State: Initialized, Role: FOLLOWER
I20260925 03:03:21.787707 24023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f58a83ec6f4149108eb0567cd7817ab5" candidate_uuid: "5dae02dd94274f54ab9c52ab57a880b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c37142ee9c043d7a5aca01030e1bbf5"
I20260925 03:03:21.789238 24023 raft_consensus.cc:3037] T f58a83ec6f4149108eb0567cd7817ab5 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:21.788975 24138 consensus_queue.cc:260] T 5055d207a0994afbbbc25a2c39455f87 P e8496e8e22c447f9b270e862bbb272a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:21.789445 24139 consensus_queue.cc:260] T 400a066077234276876ac579e8353ad2 P 1c37142ee9c043d7a5aca01030e1bbf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:21.791388 24139 ts_tablet_manager.cc:1428] T 400a066077234276876ac579e8353ad2 P 1c37142ee9c043d7a5aca01030e1bbf5: Time spent starting tablet: real 0.007s user 0.003s sys 0.001s
I20260925 03:03:21.792279 24139 tablet_bootstrap.cc:492] T 78943162e8394d039f00afadd62893f3 P 1c37142ee9c043d7a5aca01030e1bbf5: Bootstrap starting.
I20260925 03:03:21.795140 24138 ts_tablet_manager.cc:1428] T 5055d207a0994afbbbc25a2c39455f87 P e8496e8e22c447f9b270e862bbb272a3: Time spent starting tablet: real 0.011s user 0.001s sys 0.005s
I20260925 03:03:21.796300 24023 raft_consensus.cc:2454] T f58a83ec6f4149108eb0567cd7817ab5 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dae02dd94274f54ab9c52ab57a880b0 in term 1.
I20260925 03:03:21.797027 24139 tablet_bootstrap.cc:654] T 78943162e8394d039f00afadd62893f3 P 1c37142ee9c043d7a5aca01030e1bbf5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:21.789225 24142 tablet_bootstrap.cc:492] T c9f3dfa77c7845a49f80417befebb978 P 5dae02dd94274f54ab9c52ab57a880b0: No bootstrap required, opened a new log
I20260925 03:03:21.797735 24142 ts_tablet_manager.cc:1397] T c9f3dfa77c7845a49f80417befebb978 P 5dae02dd94274f54ab9c52ab57a880b0: Time spent bootstrapping tablet: real 0.146s user 0.037s sys 0.087s
I20260925 03:03:21.798146 24062 leader_election.cc:302] T f58a83ec6f4149108eb0567cd7817ab5 P 5dae02dd94274f54ab9c52ab57a880b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c37142ee9c043d7a5aca01030e1bbf5, 5dae02dd94274f54ab9c52ab57a880b0; no voters:
I20260925 03:03:21.799110 24170 raft_consensus.cc:2781] T f58a83ec6f4149108eb0567cd7817ab5 P 5dae02dd94274f54ab9c52ab57a880b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:03:21.800263 24172 leader_election.cc:288] T f58a83ec6f4149108eb0567cd7817ab5 P 5dae02dd94274f54ab9c52ab57a880b0 [CANDIDATE]: Term 1 election: Requested vote from peers 1c37142ee9c043d7a5aca01030e1bbf5 (127.22.193.2:40513), e8496e8e22c447f9b270e862bbb272a3 (127.22.193.1:44975)
I20260925 03:03:21.801956 24142 raft_consensus.cc:348] T c9f3dfa77c7845a49f80417befebb978 P 5dae02dd94274f54ab9c52ab57a880b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:21.802630 24142 raft_consensus.cc:374] T c9f3dfa77c7845a49f80417befebb978 P 5dae02dd94274f54ab9c52ab57a880b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:21.802870 24142 raft_consensus.cc:729] T c9f3dfa77c7845a49f80417befebb978 P 5dae02dd94274f54ab9c52ab57a880b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dae02dd94274f54ab9c52ab57a880b0, State: Initialized, Role: FOLLOWER
I20260925 03:03:21.803493 24142 consensus_queue.cc:260] T c9f3dfa77c7845a49f80417befebb978 P 5dae02dd94274f54ab9c52ab57a880b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:21.803851 24152 raft_consensus.cc:482] T 400a066077234276876ac579e8353ad2 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:21.804584 24152 raft_consensus.cc:504] T 400a066077234276876ac579e8353ad2 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:21.805037 24139 tablet_bootstrap.cc:492] T 78943162e8394d039f00afadd62893f3 P 1c37142ee9c043d7a5aca01030e1bbf5: No bootstrap required, opened a new log
I20260925 03:03:21.805532 24139 ts_tablet_manager.cc:1397] T 78943162e8394d039f00afadd62893f3 P 1c37142ee9c043d7a5aca01030e1bbf5: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.004s
I20260925 03:03:21.805675 24142 ts_tablet_manager.cc:1428] T c9f3dfa77c7845a49f80417befebb978 P 5dae02dd94274f54ab9c52ab57a880b0: Time spent starting tablet: real 0.008s user 0.002s sys 0.004s
I20260925 03:03:21.805999 23949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f58a83ec6f4149108eb0567cd7817ab5" candidate_uuid: "5dae02dd94274f54ab9c52ab57a880b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8496e8e22c447f9b270e862bbb272a3"
I20260925 03:03:21.806519 24142 tablet_bootstrap.cc:492] T be306de199ff4dfb976075219c02ea8e P 5dae02dd94274f54ab9c52ab57a880b0: Bootstrap starting.
I20260925 03:03:21.807071 24152 leader_election.cc:288] T 400a066077234276876ac579e8353ad2 P 1c37142ee9c043d7a5aca01030e1bbf5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e8496e8e22c447f9b270e862bbb272a3 (127.22.193.1:44975), 5dae02dd94274f54ab9c52ab57a880b0 (127.22.193.3:46383)
I20260925 03:03:21.806541 23949 raft_consensus.cc:3037] T f58a83ec6f4149108eb0567cd7817ab5 P e8496e8e22c447f9b270e862bbb272a3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:21.808538 23948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "400a066077234276876ac579e8353ad2" candidate_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8496e8e22c447f9b270e862bbb272a3" is_pre_election: true
I20260925 03:03:21.809180 23948 raft_consensus.cc:2454] T 400a066077234276876ac579e8353ad2 P e8496e8e22c447f9b270e862bbb272a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c37142ee9c043d7a5aca01030e1bbf5 in term 0.
I20260925 03:03:21.810283 23988 leader_election.cc:302] T 400a066077234276876ac579e8353ad2 P 1c37142ee9c043d7a5aca01030e1bbf5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c37142ee9c043d7a5aca01030e1bbf5, e8496e8e22c447f9b270e862bbb272a3; no voters:
I20260925 03:03:21.810808 24097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "400a066077234276876ac579e8353ad2" candidate_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dae02dd94274f54ab9c52ab57a880b0" is_pre_election: true
I20260925 03:03:21.811072 24139 raft_consensus.cc:348] T 78943162e8394d039f00afadd62893f3 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:21.811443 24097 raft_consensus.cc:2454] T 400a066077234276876ac579e8353ad2 P 5dae02dd94274f54ab9c52ab57a880b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c37142ee9c043d7a5aca01030e1bbf5 in term 0.
I20260925 03:03:21.811749 24139 raft_consensus.cc:374] T 78943162e8394d039f00afadd62893f3 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:21.811837 24152 raft_consensus.cc:2781] T 400a066077234276876ac579e8353ad2 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 03:03:21.812093 24139 raft_consensus.cc:729] T 78943162e8394d039f00afadd62893f3 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c37142ee9c043d7a5aca01030e1bbf5, State: Initialized, Role: FOLLOWER
I20260925 03:03:21.812443 24142 tablet_bootstrap.cc:654] T be306de199ff4dfb976075219c02ea8e P 5dae02dd94274f54ab9c52ab57a880b0: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:21.812790 24139 consensus_queue.cc:260] T 78943162e8394d039f00afadd62893f3 P 1c37142ee9c043d7a5aca01030e1bbf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:21.814095 24143 raft_consensus.cc:482] T 5055d207a0994afbbbc25a2c39455f87 P e8496e8e22c447f9b270e862bbb272a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:21.814524 24139 ts_tablet_manager.cc:1428] T 78943162e8394d039f00afadd62893f3 P 1c37142ee9c043d7a5aca01030e1bbf5: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260925 03:03:21.814603 24143 raft_consensus.cc:504] T 5055d207a0994afbbbc25a2c39455f87 P e8496e8e22c447f9b270e862bbb272a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:21.815549 24139 tablet_bootstrap.cc:492] T 48c4fddba997473f8a501507277b8860 P 1c37142ee9c043d7a5aca01030e1bbf5: Bootstrap starting.
I20260925 03:03:21.812345 24152 raft_consensus.cc:482] T 400a066077234276876ac579e8353ad2 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:03:21.817395 24152 raft_consensus.cc:3037] T 400a066077234276876ac579e8353ad2 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:21.817914 24023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5055d207a0994afbbbc25a2c39455f87" candidate_uuid: "e8496e8e22c447f9b270e862bbb272a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" is_pre_election: true
I20260925 03:03:21.818572 24023 raft_consensus.cc:2454] T 5055d207a0994afbbbc25a2c39455f87 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8496e8e22c447f9b270e862bbb272a3 in term 0.
I20260925 03:03:21.818711 24097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5055d207a0994afbbbc25a2c39455f87" candidate_uuid: "e8496e8e22c447f9b270e862bbb272a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dae02dd94274f54ab9c52ab57a880b0" is_pre_election: true
I20260925 03:03:21.819413 24097 raft_consensus.cc:2454] T 5055d207a0994afbbbc25a2c39455f87 P 5dae02dd94274f54ab9c52ab57a880b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8496e8e22c447f9b270e862bbb272a3 in term 0.
I20260925 03:03:21.819859 23913 leader_election.cc:302] T 5055d207a0994afbbbc25a2c39455f87 P e8496e8e22c447f9b270e862bbb272a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c37142ee9c043d7a5aca01030e1bbf5, e8496e8e22c447f9b270e862bbb272a3; no voters:
I20260925 03:03:21.821141 24180 raft_consensus.cc:2781] T 5055d207a0994afbbbc25a2c39455f87 P e8496e8e22c447f9b270e862bbb272a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 03:03:21.821303 24139 tablet_bootstrap.cc:654] T 48c4fddba997473f8a501507277b8860 P 1c37142ee9c043d7a5aca01030e1bbf5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:21.821570 24180 raft_consensus.cc:482] T 5055d207a0994afbbbc25a2c39455f87 P e8496e8e22c447f9b270e862bbb272a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:03:21.821916 24180 raft_consensus.cc:3037] T 5055d207a0994afbbbc25a2c39455f87 P e8496e8e22c447f9b270e862bbb272a3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:21.821949 24143 leader_election.cc:288] T 5055d207a0994afbbbc25a2c39455f87 P e8496e8e22c447f9b270e862bbb272a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c37142ee9c043d7a5aca01030e1bbf5 (127.22.193.2:40513), 5dae02dd94274f54ab9c52ab57a880b0 (127.22.193.3:46383)
I20260925 03:03:21.828939 24180 raft_consensus.cc:504] T 5055d207a0994afbbbc25a2c39455f87 P e8496e8e22c447f9b270e862bbb272a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:21.830228 23847 catalog_manager.cc:5495] T 845902c057d54c5ab9ddc861f622125c P 5dae02dd94274f54ab9c52ab57a880b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5dae02dd94274f54ab9c52ab57a880b0 (127.22.193.3). New cstate: current_term: 1 leader_uuid: "5dae02dd94274f54ab9c52ab57a880b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } health_report { overall_health: HEALTHY } } }
I20260925 03:03:21.832135 24023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5055d207a0994afbbbc25a2c39455f87" candidate_uuid: "e8496e8e22c447f9b270e862bbb272a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c37142ee9c043d7a5aca01030e1bbf5"
I20260925 03:03:21.833590 24023 raft_consensus.cc:3037] T 5055d207a0994afbbbc25a2c39455f87 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:21.834362 24097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5055d207a0994afbbbc25a2c39455f87" candidate_uuid: "e8496e8e22c447f9b270e862bbb272a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dae02dd94274f54ab9c52ab57a880b0"
I20260925 03:03:21.835006 24097 raft_consensus.cc:3037] T 5055d207a0994afbbbc25a2c39455f87 P 5dae02dd94274f54ab9c52ab57a880b0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:21.832614 24152 raft_consensus.cc:504] T 400a066077234276876ac579e8353ad2 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:21.833091 24180 leader_election.cc:288] T 5055d207a0994afbbbc25a2c39455f87 P e8496e8e22c447f9b270e862bbb272a3 [CANDIDATE]: Term 1 election: Requested vote from peers 1c37142ee9c043d7a5aca01030e1bbf5 (127.22.193.2:40513), 5dae02dd94274f54ab9c52ab57a880b0 (127.22.193.3:46383)
I20260925 03:03:21.845536 24023 raft_consensus.cc:2454] T 5055d207a0994afbbbc25a2c39455f87 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8496e8e22c447f9b270e862bbb272a3 in term 1.
I20260925 03:03:21.846781 24152 leader_election.cc:288] T 400a066077234276876ac579e8353ad2 P 1c37142ee9c043d7a5aca01030e1bbf5 [CANDIDATE]: Term 1 election: Requested vote from peers e8496e8e22c447f9b270e862bbb272a3 (127.22.193.1:44975), 5dae02dd94274f54ab9c52ab57a880b0 (127.22.193.3:46383)
I20260925 03:03:21.847800 23948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "400a066077234276876ac579e8353ad2" candidate_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8496e8e22c447f9b270e862bbb272a3"
I20260925 03:03:21.848456 23948 raft_consensus.cc:3037] T 400a066077234276876ac579e8353ad2 P e8496e8e22c447f9b270e862bbb272a3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:21.850576 24139 tablet_bootstrap.cc:492] T 48c4fddba997473f8a501507277b8860 P 1c37142ee9c043d7a5aca01030e1bbf5: No bootstrap required, opened a new log
I20260925 03:03:21.851028 24139 ts_tablet_manager.cc:1397] T 48c4fddba997473f8a501507277b8860 P 1c37142ee9c043d7a5aca01030e1bbf5: Time spent bootstrapping tablet: real 0.036s user 0.024s sys 0.008s
I20260925 03:03:21.853315 23949 raft_consensus.cc:2454] T f58a83ec6f4149108eb0567cd7817ab5 P e8496e8e22c447f9b270e862bbb272a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dae02dd94274f54ab9c52ab57a880b0 in term 1.
I20260925 03:03:21.853430 23913 leader_election.cc:302] T 5055d207a0994afbbbc25a2c39455f87 P e8496e8e22c447f9b270e862bbb272a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c37142ee9c043d7a5aca01030e1bbf5, e8496e8e22c447f9b270e862bbb272a3; no voters:
I20260925 03:03:21.856992 24180 raft_consensus.cc:2781] T 5055d207a0994afbbbc25a2c39455f87 P e8496e8e22c447f9b270e862bbb272a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:03:21.857438 24180 raft_consensus.cc:686] T 5055d207a0994afbbbc25a2c39455f87 P e8496e8e22c447f9b270e862bbb272a3 [term 1 LEADER]: Becoming Leader. State: Replica: e8496e8e22c447f9b270e862bbb272a3, State: Running, Role: LEADER
I20260925 03:03:21.857993 23948 raft_consensus.cc:2454] T 400a066077234276876ac579e8353ad2 P e8496e8e22c447f9b270e862bbb272a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c37142ee9c043d7a5aca01030e1bbf5 in term 1.
I20260925 03:03:21.858176 24180 consensus_queue.cc:237] T 5055d207a0994afbbbc25a2c39455f87 P e8496e8e22c447f9b270e862bbb272a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:21.858884 23988 leader_election.cc:302] T 400a066077234276876ac579e8353ad2 P 1c37142ee9c043d7a5aca01030e1bbf5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c37142ee9c043d7a5aca01030e1bbf5, e8496e8e22c447f9b270e862bbb272a3; no voters:
I20260925 03:03:21.859661 24152 raft_consensus.cc:2781] T 400a066077234276876ac579e8353ad2 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:03:21.860116 24152 raft_consensus.cc:686] T 400a066077234276876ac579e8353ad2 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 1 LEADER]: Becoming Leader. State: Replica: 1c37142ee9c043d7a5aca01030e1bbf5, State: Running, Role: LEADER
I20260925 03:03:21.860893 24152 consensus_queue.cc:237] T 400a066077234276876ac579e8353ad2 P 1c37142ee9c043d7a5aca01030e1bbf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:21.843127 24097 raft_consensus.cc:2454] T 5055d207a0994afbbbc25a2c39455f87 P 5dae02dd94274f54ab9c52ab57a880b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8496e8e22c447f9b270e862bbb272a3 in term 1.
I20260925 03:03:21.853667 24139 raft_consensus.cc:348] T 48c4fddba997473f8a501507277b8860 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:21.855895 24098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "400a066077234276876ac579e8353ad2" candidate_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dae02dd94274f54ab9c52ab57a880b0"
I20260925 03:03:21.863020 24139 raft_consensus.cc:374] T 48c4fddba997473f8a501507277b8860 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:21.863070 24098 raft_consensus.cc:3037] T 400a066077234276876ac579e8353ad2 P 5dae02dd94274f54ab9c52ab57a880b0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:21.863416 24139 raft_consensus.cc:729] T 48c4fddba997473f8a501507277b8860 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c37142ee9c043d7a5aca01030e1bbf5, State: Initialized, Role: FOLLOWER
I20260925 03:03:21.864274 24139 consensus_queue.cc:260] T 48c4fddba997473f8a501507277b8860 P 1c37142ee9c043d7a5aca01030e1bbf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:21.866586 24139 ts_tablet_manager.cc:1428] T 48c4fddba997473f8a501507277b8860 P 1c37142ee9c043d7a5aca01030e1bbf5: Time spent starting tablet: real 0.015s user 0.007s sys 0.000s
I20260925 03:03:21.856010 24170 raft_consensus.cc:686] T f58a83ec6f4149108eb0567cd7817ab5 P 5dae02dd94274f54ab9c52ab57a880b0 [term 1 LEADER]: Becoming Leader. State: Replica: 5dae02dd94274f54ab9c52ab57a880b0, State: Running, Role: LEADER
I20260925 03:03:21.869488 24170 consensus_queue.cc:237] T f58a83ec6f4149108eb0567cd7817ab5 P 5dae02dd94274f54ab9c52ab57a880b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:21.870206 24098 raft_consensus.cc:2454] T 400a066077234276876ac579e8353ad2 P 5dae02dd94274f54ab9c52ab57a880b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c37142ee9c043d7a5aca01030e1bbf5 in term 1.
I20260925 03:03:21.869879 23845 catalog_manager.cc:5495] T 5055d207a0994afbbbc25a2c39455f87 P e8496e8e22c447f9b270e862bbb272a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to e8496e8e22c447f9b270e862bbb272a3 (127.22.193.1). New cstate: current_term: 1 leader_uuid: "e8496e8e22c447f9b270e862bbb272a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } health_report { overall_health: UNKNOWN } } }
I20260925 03:03:21.875320 24142 tablet_bootstrap.cc:492] T be306de199ff4dfb976075219c02ea8e P 5dae02dd94274f54ab9c52ab57a880b0: No bootstrap required, opened a new log
I20260925 03:03:21.875982 24142 ts_tablet_manager.cc:1397] T be306de199ff4dfb976075219c02ea8e P 5dae02dd94274f54ab9c52ab57a880b0: Time spent bootstrapping tablet: real 0.070s user 0.020s sys 0.010s
I20260925 03:03:21.878839 24142 raft_consensus.cc:348] T be306de199ff4dfb976075219c02ea8e P 5dae02dd94274f54ab9c52ab57a880b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:21.879999 24142 raft_consensus.cc:374] T be306de199ff4dfb976075219c02ea8e P 5dae02dd94274f54ab9c52ab57a880b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:21.880290 24142 raft_consensus.cc:729] T be306de199ff4dfb976075219c02ea8e P 5dae02dd94274f54ab9c52ab57a880b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dae02dd94274f54ab9c52ab57a880b0, State: Initialized, Role: FOLLOWER
I20260925 03:03:21.880955 24142 consensus_queue.cc:260] T be306de199ff4dfb976075219c02ea8e P 5dae02dd94274f54ab9c52ab57a880b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:21.882604 24142 ts_tablet_manager.cc:1428] T be306de199ff4dfb976075219c02ea8e P 5dae02dd94274f54ab9c52ab57a880b0: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260925 03:03:21.883278 24142 tablet_bootstrap.cc:492] T 78943162e8394d039f00afadd62893f3 P 5dae02dd94274f54ab9c52ab57a880b0: Bootstrap starting.
I20260925 03:03:21.886096 23845 catalog_manager.cc:5495] T f58a83ec6f4149108eb0567cd7817ab5 P 5dae02dd94274f54ab9c52ab57a880b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5dae02dd94274f54ab9c52ab57a880b0 (127.22.193.3). New cstate: current_term: 1 leader_uuid: "5dae02dd94274f54ab9c52ab57a880b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } health_report { overall_health: HEALTHY } } }
I20260925 03:03:21.886251 23847 catalog_manager.cc:5495] T 400a066077234276876ac579e8353ad2 P 1c37142ee9c043d7a5aca01030e1bbf5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c37142ee9c043d7a5aca01030e1bbf5 (127.22.193.2). New cstate: current_term: 1 leader_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } health_report { overall_health: UNKNOWN } } }
I20260925 03:03:21.888410 24142 tablet_bootstrap.cc:654] T 78943162e8394d039f00afadd62893f3 P 5dae02dd94274f54ab9c52ab57a880b0: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:21.894981 24142 tablet_bootstrap.cc:492] T 78943162e8394d039f00afadd62893f3 P 5dae02dd94274f54ab9c52ab57a880b0: No bootstrap required, opened a new log
I20260925 03:03:21.895311 24142 ts_tablet_manager.cc:1397] T 78943162e8394d039f00afadd62893f3 P 5dae02dd94274f54ab9c52ab57a880b0: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20260925 03:03:21.897682 24142 raft_consensus.cc:348] T 78943162e8394d039f00afadd62893f3 P 5dae02dd94274f54ab9c52ab57a880b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:21.898375 24142 raft_consensus.cc:374] T 78943162e8394d039f00afadd62893f3 P 5dae02dd94274f54ab9c52ab57a880b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:21.898639 24142 raft_consensus.cc:729] T 78943162e8394d039f00afadd62893f3 P 5dae02dd94274f54ab9c52ab57a880b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dae02dd94274f54ab9c52ab57a880b0, State: Initialized, Role: FOLLOWER
I20260925 03:03:21.899222 24142 consensus_queue.cc:260] T 78943162e8394d039f00afadd62893f3 P 5dae02dd94274f54ab9c52ab57a880b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:21.900978 24142 ts_tablet_manager.cc:1428] T 78943162e8394d039f00afadd62893f3 P 5dae02dd94274f54ab9c52ab57a880b0: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260925 03:03:21.912725 24170 raft_consensus.cc:482] T 78943162e8394d039f00afadd62893f3 P 5dae02dd94274f54ab9c52ab57a880b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:21.913184 24170 raft_consensus.cc:504] T 78943162e8394d039f00afadd62893f3 P 5dae02dd94274f54ab9c52ab57a880b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:21.914755 24170 leader_election.cc:288] T 78943162e8394d039f00afadd62893f3 P 5dae02dd94274f54ab9c52ab57a880b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e8496e8e22c447f9b270e862bbb272a3 (127.22.193.1:44975), 1c37142ee9c043d7a5aca01030e1bbf5 (127.22.193.2:40513)
I20260925 03:03:21.915699 23949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78943162e8394d039f00afadd62893f3" candidate_uuid: "5dae02dd94274f54ab9c52ab57a880b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8496e8e22c447f9b270e862bbb272a3" is_pre_election: true
I20260925 03:03:21.915853 24023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78943162e8394d039f00afadd62893f3" candidate_uuid: "5dae02dd94274f54ab9c52ab57a880b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" is_pre_election: true
I20260925 03:03:21.916402 24023 raft_consensus.cc:2454] T 78943162e8394d039f00afadd62893f3 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dae02dd94274f54ab9c52ab57a880b0 in term 0.
I20260925 03:03:21.916410 23949 raft_consensus.cc:2454] T 78943162e8394d039f00afadd62893f3 P e8496e8e22c447f9b270e862bbb272a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dae02dd94274f54ab9c52ab57a880b0 in term 0.
I20260925 03:03:21.917472 24062 leader_election.cc:302] T 78943162e8394d039f00afadd62893f3 P 5dae02dd94274f54ab9c52ab57a880b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c37142ee9c043d7a5aca01030e1bbf5, 5dae02dd94274f54ab9c52ab57a880b0; no voters:
I20260925 03:03:21.918097 24170 raft_consensus.cc:2781] T 78943162e8394d039f00afadd62893f3 P 5dae02dd94274f54ab9c52ab57a880b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 03:03:21.918438 24170 raft_consensus.cc:482] T 78943162e8394d039f00afadd62893f3 P 5dae02dd94274f54ab9c52ab57a880b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:03:21.918715 24170 raft_consensus.cc:3037] T 78943162e8394d039f00afadd62893f3 P 5dae02dd94274f54ab9c52ab57a880b0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:21.923272 24170 raft_consensus.cc:504] T 78943162e8394d039f00afadd62893f3 P 5dae02dd94274f54ab9c52ab57a880b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:21.924849 24170 leader_election.cc:288] T 78943162e8394d039f00afadd62893f3 P 5dae02dd94274f54ab9c52ab57a880b0 [CANDIDATE]: Term 1 election: Requested vote from peers e8496e8e22c447f9b270e862bbb272a3 (127.22.193.1:44975), 1c37142ee9c043d7a5aca01030e1bbf5 (127.22.193.2:40513)
I20260925 03:03:21.925802 23949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78943162e8394d039f00afadd62893f3" candidate_uuid: "5dae02dd94274f54ab9c52ab57a880b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8496e8e22c447f9b270e862bbb272a3"
I20260925 03:03:21.926000 24023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78943162e8394d039f00afadd62893f3" candidate_uuid: "5dae02dd94274f54ab9c52ab57a880b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c37142ee9c043d7a5aca01030e1bbf5"
I20260925 03:03:21.926429 23949 raft_consensus.cc:3037] T 78943162e8394d039f00afadd62893f3 P e8496e8e22c447f9b270e862bbb272a3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:21.926518 24023 raft_consensus.cc:3037] T 78943162e8394d039f00afadd62893f3 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:21.931277 23949 raft_consensus.cc:2454] T 78943162e8394d039f00afadd62893f3 P e8496e8e22c447f9b270e862bbb272a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dae02dd94274f54ab9c52ab57a880b0 in term 1.
I20260925 03:03:21.932240 24023 raft_consensus.cc:2454] T 78943162e8394d039f00afadd62893f3 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dae02dd94274f54ab9c52ab57a880b0 in term 1.
I20260925 03:03:21.932258 24062 leader_election.cc:302] T 78943162e8394d039f00afadd62893f3 P 5dae02dd94274f54ab9c52ab57a880b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dae02dd94274f54ab9c52ab57a880b0, e8496e8e22c447f9b270e862bbb272a3; no voters:
I20260925 03:03:21.933151 24170 raft_consensus.cc:2781] T 78943162e8394d039f00afadd62893f3 P 5dae02dd94274f54ab9c52ab57a880b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:03:21.933531 24170 raft_consensus.cc:686] T 78943162e8394d039f00afadd62893f3 P 5dae02dd94274f54ab9c52ab57a880b0 [term 1 LEADER]: Becoming Leader. State: Replica: 5dae02dd94274f54ab9c52ab57a880b0, State: Running, Role: LEADER
I20260925 03:03:21.934281 24170 consensus_queue.cc:237] T 78943162e8394d039f00afadd62893f3 P 5dae02dd94274f54ab9c52ab57a880b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:21.941455 23845 catalog_manager.cc:5495] T 78943162e8394d039f00afadd62893f3 P 5dae02dd94274f54ab9c52ab57a880b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5dae02dd94274f54ab9c52ab57a880b0 (127.22.193.3). New cstate: current_term: 1 leader_uuid: "5dae02dd94274f54ab9c52ab57a880b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } health_report { overall_health: HEALTHY } } }
I20260925 03:03:21.950608 24170 consensus_queue.cc:1035] T 5ac7083552944834bb61700cfbe60faa P 5dae02dd94274f54ab9c52ab57a880b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 03:03:21.963299 24190 consensus_queue.cc:1035] T 5ac7083552944834bb61700cfbe60faa P 5dae02dd94274f54ab9c52ab57a880b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260925 03:03:21.977000 24190 raft_consensus.cc:482] T c9f3dfa77c7845a49f80417befebb978 P 5dae02dd94274f54ab9c52ab57a880b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:21.977502 24190 raft_consensus.cc:504] T c9f3dfa77c7845a49f80417befebb978 P 5dae02dd94274f54ab9c52ab57a880b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:21.979303 24190 leader_election.cc:288] T c9f3dfa77c7845a49f80417befebb978 P 5dae02dd94274f54ab9c52ab57a880b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e8496e8e22c447f9b270e862bbb272a3 (127.22.193.1:44975), 1c37142ee9c043d7a5aca01030e1bbf5 (127.22.193.2:40513)
I20260925 03:03:21.980023 23949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9f3dfa77c7845a49f80417befebb978" candidate_uuid: "5dae02dd94274f54ab9c52ab57a880b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8496e8e22c447f9b270e862bbb272a3" is_pre_election: true
I20260925 03:03:21.980211 24023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9f3dfa77c7845a49f80417befebb978" candidate_uuid: "5dae02dd94274f54ab9c52ab57a880b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" is_pre_election: true
I20260925 03:03:21.980890 23949 raft_consensus.cc:2379] T c9f3dfa77c7845a49f80417befebb978 P e8496e8e22c447f9b270e862bbb272a3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5dae02dd94274f54ab9c52ab57a880b0 in current term 1: Already voted for candidate 1c37142ee9c043d7a5aca01030e1bbf5 in this term.
I20260925 03:03:21.981712 24062 leader_election.cc:302] T c9f3dfa77c7845a49f80417befebb978 P 5dae02dd94274f54ab9c52ab57a880b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5dae02dd94274f54ab9c52ab57a880b0; no voters: 1c37142ee9c043d7a5aca01030e1bbf5, e8496e8e22c447f9b270e862bbb272a3
I20260925 03:03:21.982357 24190 raft_consensus.cc:3037] T c9f3dfa77c7845a49f80417befebb978 P 5dae02dd94274f54ab9c52ab57a880b0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:21.988498 24190 raft_consensus.cc:2726] T c9f3dfa77c7845a49f80417befebb978 P 5dae02dd94274f54ab9c52ab57a880b0 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 03:03:21.988628 24152 consensus_queue.cc:1035] T c9f3dfa77c7845a49f80417befebb978 P 1c37142ee9c043d7a5aca01030e1bbf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 03:03:21.989948 24152 consensus_queue.cc:1035] T 7f1e503d876c45fa853edeaf3b836180 P 1c37142ee9c043d7a5aca01030e1bbf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 03:03:21.999877 24098 raft_consensus.cc:3037] T 7f1e503d876c45fa853edeaf3b836180 P 5dae02dd94274f54ab9c52ab57a880b0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:21.999784 24154 consensus_queue.cc:1035] T c9f3dfa77c7845a49f80417befebb978 P 1c37142ee9c043d7a5aca01030e1bbf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 03:03:22.003770 24149 raft_consensus.cc:482] T be306de199ff4dfb976075219c02ea8e P 5dae02dd94274f54ab9c52ab57a880b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:22.004246 24149 raft_consensus.cc:504] T be306de199ff4dfb976075219c02ea8e P 5dae02dd94274f54ab9c52ab57a880b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:22.006292 24149 leader_election.cc:288] T be306de199ff4dfb976075219c02ea8e P 5dae02dd94274f54ab9c52ab57a880b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e8496e8e22c447f9b270e862bbb272a3 (127.22.193.1:44975), 1c37142ee9c043d7a5aca01030e1bbf5 (127.22.193.2:40513)
I20260925 03:03:22.009083 24023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be306de199ff4dfb976075219c02ea8e" candidate_uuid: "5dae02dd94274f54ab9c52ab57a880b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" is_pre_election: true
I20260925 03:03:22.010874 23948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be306de199ff4dfb976075219c02ea8e" candidate_uuid: "5dae02dd94274f54ab9c52ab57a880b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8496e8e22c447f9b270e862bbb272a3" is_pre_election: true
I20260925 03:03:22.011655 23948 raft_consensus.cc:2379] T be306de199ff4dfb976075219c02ea8e P e8496e8e22c447f9b270e862bbb272a3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5dae02dd94274f54ab9c52ab57a880b0 in current term 1: Already voted for candidate 1c37142ee9c043d7a5aca01030e1bbf5 in this term.
I20260925 03:03:22.012801 24062 leader_election.cc:302] T be306de199ff4dfb976075219c02ea8e P 5dae02dd94274f54ab9c52ab57a880b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5dae02dd94274f54ab9c52ab57a880b0; no voters: 1c37142ee9c043d7a5aca01030e1bbf5, e8496e8e22c447f9b270e862bbb272a3
I20260925 03:03:22.012912 24154 consensus_queue.cc:1035] T 7f1e503d876c45fa853edeaf3b836180 P 1c37142ee9c043d7a5aca01030e1bbf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260925 03:03:22.014171 24149 raft_consensus.cc:3037] T be306de199ff4dfb976075219c02ea8e P 5dae02dd94274f54ab9c52ab57a880b0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:22.021420 24149 raft_consensus.cc:2726] T be306de199ff4dfb976075219c02ea8e P 5dae02dd94274f54ab9c52ab57a880b0 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 03:03:22.037001 24149 raft_consensus.cc:482] T 48c4fddba997473f8a501507277b8860 P 5dae02dd94274f54ab9c52ab57a880b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:22.037487 24149 raft_consensus.cc:504] T 48c4fddba997473f8a501507277b8860 P 5dae02dd94274f54ab9c52ab57a880b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:22.039083 24149 leader_election.cc:288] T 48c4fddba997473f8a501507277b8860 P 5dae02dd94274f54ab9c52ab57a880b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e8496e8e22c447f9b270e862bbb272a3 (127.22.193.1:44975), 1c37142ee9c043d7a5aca01030e1bbf5 (127.22.193.2:40513)
I20260925 03:03:22.039949 23948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48c4fddba997473f8a501507277b8860" candidate_uuid: "5dae02dd94274f54ab9c52ab57a880b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8496e8e22c447f9b270e862bbb272a3" is_pre_election: true
I20260925 03:03:22.040268 24023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48c4fddba997473f8a501507277b8860" candidate_uuid: "5dae02dd94274f54ab9c52ab57a880b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" is_pre_election: true
I20260925 03:03:22.040676 23948 raft_consensus.cc:2454] T 48c4fddba997473f8a501507277b8860 P e8496e8e22c447f9b270e862bbb272a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dae02dd94274f54ab9c52ab57a880b0 in term 0.
I20260925 03:03:22.040923 24023 raft_consensus.cc:2454] T 48c4fddba997473f8a501507277b8860 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dae02dd94274f54ab9c52ab57a880b0 in term 0.
I20260925 03:03:22.042012 24062 leader_election.cc:302] T 48c4fddba997473f8a501507277b8860 P 5dae02dd94274f54ab9c52ab57a880b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dae02dd94274f54ab9c52ab57a880b0, e8496e8e22c447f9b270e862bbb272a3; no voters:
I20260925 03:03:22.042671 24149 raft_consensus.cc:2781] T 48c4fddba997473f8a501507277b8860 P 5dae02dd94274f54ab9c52ab57a880b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 03:03:22.042961 24149 raft_consensus.cc:482] T 48c4fddba997473f8a501507277b8860 P 5dae02dd94274f54ab9c52ab57a880b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:03:22.043237 24149 raft_consensus.cc:3037] T 48c4fddba997473f8a501507277b8860 P 5dae02dd94274f54ab9c52ab57a880b0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:22.047598 24149 raft_consensus.cc:504] T 48c4fddba997473f8a501507277b8860 P 5dae02dd94274f54ab9c52ab57a880b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:22.049151 24149 leader_election.cc:288] T 48c4fddba997473f8a501507277b8860 P 5dae02dd94274f54ab9c52ab57a880b0 [CANDIDATE]: Term 1 election: Requested vote from peers e8496e8e22c447f9b270e862bbb272a3 (127.22.193.1:44975), 1c37142ee9c043d7a5aca01030e1bbf5 (127.22.193.2:40513)
I20260925 03:03:22.050015 23948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48c4fddba997473f8a501507277b8860" candidate_uuid: "5dae02dd94274f54ab9c52ab57a880b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8496e8e22c447f9b270e862bbb272a3"
I20260925 03:03:22.050132 24023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48c4fddba997473f8a501507277b8860" candidate_uuid: "5dae02dd94274f54ab9c52ab57a880b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c37142ee9c043d7a5aca01030e1bbf5"
I20260925 03:03:22.050534 23948 raft_consensus.cc:3037] T 48c4fddba997473f8a501507277b8860 P e8496e8e22c447f9b270e862bbb272a3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:22.050602 24023 raft_consensus.cc:3037] T 48c4fddba997473f8a501507277b8860 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:22.055347 24023 raft_consensus.cc:2454] T 48c4fddba997473f8a501507277b8860 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dae02dd94274f54ab9c52ab57a880b0 in term 1.
I20260925 03:03:22.055447 23948 raft_consensus.cc:2454] T 48c4fddba997473f8a501507277b8860 P e8496e8e22c447f9b270e862bbb272a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dae02dd94274f54ab9c52ab57a880b0 in term 1.
I20260925 03:03:22.056296 24062 leader_election.cc:302] T 48c4fddba997473f8a501507277b8860 P 5dae02dd94274f54ab9c52ab57a880b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c37142ee9c043d7a5aca01030e1bbf5, 5dae02dd94274f54ab9c52ab57a880b0; no voters:
I20260925 03:03:22.056969 24149 raft_consensus.cc:2781] T 48c4fddba997473f8a501507277b8860 P 5dae02dd94274f54ab9c52ab57a880b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:03:22.057426 24149 raft_consensus.cc:686] T 48c4fddba997473f8a501507277b8860 P 5dae02dd94274f54ab9c52ab57a880b0 [term 1 LEADER]: Becoming Leader. State: Replica: 5dae02dd94274f54ab9c52ab57a880b0, State: Running, Role: LEADER
I20260925 03:03:22.058007 24149 consensus_queue.cc:237] T 48c4fddba997473f8a501507277b8860 P 5dae02dd94274f54ab9c52ab57a880b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:22.064963 23845 catalog_manager.cc:5495] T 48c4fddba997473f8a501507277b8860 P 5dae02dd94274f54ab9c52ab57a880b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5dae02dd94274f54ab9c52ab57a880b0 (127.22.193.3). New cstate: current_term: 1 leader_uuid: "5dae02dd94274f54ab9c52ab57a880b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } health_report { overall_health: HEALTHY } } }
I20260925 03:03:22.089193 24154 consensus_queue.cc:1035] T be306de199ff4dfb976075219c02ea8e P 1c37142ee9c043d7a5aca01030e1bbf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 03:03:22.103724 24164 consensus_queue.cc:1035] T be306de199ff4dfb976075219c02ea8e P 1c37142ee9c043d7a5aca01030e1bbf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 03:03:22.188364 24149 consensus_queue.cc:1035] T 845902c057d54c5ab9ddc861f622125c P 5dae02dd94274f54ab9c52ab57a880b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 03:03:22.197350 24149 consensus_queue.cc:1035] T 845902c057d54c5ab9ddc861f622125c P 5dae02dd94274f54ab9c52ab57a880b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 03:03:22.221422 24098 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f58a83ec6f4149108eb0567cd7817ab5"
dest_uuid: "5dae02dd94274f54ab9c52ab57a880b0"
mode: GRACEFUL
new_leader_uuid: "e8496e8e22c447f9b270e862bbb272a3"
from {username='slave'} at 127.0.0.1:46502
I20260925 03:03:22.222081 24098 raft_consensus.cc:595] T f58a83ec6f4149108eb0567cd7817ab5 P 5dae02dd94274f54ab9c52ab57a880b0 [term 1 LEADER]: Received request to transfer leadership to TS e8496e8e22c447f9b270e862bbb272a3
I20260925 03:03:22.236120 24023 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7f1e503d876c45fa853edeaf3b836180"
dest_uuid: "1c37142ee9c043d7a5aca01030e1bbf5"
mode: GRACEFUL
new_leader_uuid: "e8496e8e22c447f9b270e862bbb272a3"
from {username='slave'} at 127.0.0.1:40134
I20260925 03:03:22.236778 24023 raft_consensus.cc:595] T 7f1e503d876c45fa853edeaf3b836180 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 1 LEADER]: Received request to transfer leadership to TS e8496e8e22c447f9b270e862bbb272a3
I20260925 03:03:22.241645 24023 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c9f3dfa77c7845a49f80417befebb978"
dest_uuid: "1c37142ee9c043d7a5aca01030e1bbf5"
mode: GRACEFUL
new_leader_uuid: "e8496e8e22c447f9b270e862bbb272a3"
from {username='slave'} at 127.0.0.1:40134
I20260925 03:03:22.242161 24023 raft_consensus.cc:595] T c9f3dfa77c7845a49f80417befebb978 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 1 LEADER]: Received request to transfer leadership to TS e8496e8e22c447f9b270e862bbb272a3
I20260925 03:03:22.246977 24098 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "48c4fddba997473f8a501507277b8860"
dest_uuid: "5dae02dd94274f54ab9c52ab57a880b0"
mode: GRACEFUL
new_leader_uuid: "e8496e8e22c447f9b270e862bbb272a3"
from {username='slave'} at 127.0.0.1:46502
I20260925 03:03:22.247519 24098 raft_consensus.cc:595] T 48c4fddba997473f8a501507277b8860 P 5dae02dd94274f54ab9c52ab57a880b0 [term 1 LEADER]: Received request to transfer leadership to TS e8496e8e22c447f9b270e862bbb272a3
I20260925 03:03:22.252835 24098 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "845902c057d54c5ab9ddc861f622125c"
dest_uuid: "5dae02dd94274f54ab9c52ab57a880b0"
mode: GRACEFUL
new_leader_uuid: "e8496e8e22c447f9b270e862bbb272a3"
from {username='slave'} at 127.0.0.1:46502
I20260925 03:03:22.253103 24164 consensus_queue.cc:1035] T 400a066077234276876ac579e8353ad2 P 1c37142ee9c043d7a5aca01030e1bbf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 03:03:22.253399 24098 raft_consensus.cc:595] T 845902c057d54c5ab9ddc861f622125c P 5dae02dd94274f54ab9c52ab57a880b0 [term 1 LEADER]: Received request to transfer leadership to TS e8496e8e22c447f9b270e862bbb272a3
I20260925 03:03:22.262059 24098 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "78943162e8394d039f00afadd62893f3"
dest_uuid: "5dae02dd94274f54ab9c52ab57a880b0"
mode: GRACEFUL
new_leader_uuid: "1c37142ee9c043d7a5aca01030e1bbf5"
from {username='slave'} at 127.0.0.1:46502
I20260925 03:03:22.262630 24098 raft_consensus.cc:595] T 78943162e8394d039f00afadd62893f3 P 5dae02dd94274f54ab9c52ab57a880b0 [term 1 LEADER]: Received request to transfer leadership to TS 1c37142ee9c043d7a5aca01030e1bbf5
I20260925 03:03:22.265292 24152 consensus_queue.cc:1035] T 400a066077234276876ac579e8353ad2 P 1c37142ee9c043d7a5aca01030e1bbf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 03:03:22.293128 23948 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5055d207a0994afbbbc25a2c39455f87"
dest_uuid: "e8496e8e22c447f9b270e862bbb272a3"
mode: GRACEFUL
new_leader_uuid: "1c37142ee9c043d7a5aca01030e1bbf5"
from {username='slave'} at 127.0.0.1:50470
I20260925 03:03:22.293777 23948 raft_consensus.cc:595] T 5055d207a0994afbbbc25a2c39455f87 P e8496e8e22c447f9b270e862bbb272a3 [term 1 LEADER]: Received request to transfer leadership to TS 1c37142ee9c043d7a5aca01030e1bbf5
I20260925 03:03:22.320217 24149 consensus_queue.cc:1035] T 78943162e8394d039f00afadd62893f3 P 5dae02dd94274f54ab9c52ab57a880b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 03:03:22.327080 24170 raft_consensus.cc:979] T 78943162e8394d039f00afadd62893f3 P 5dae02dd94274f54ab9c52ab57a880b0: : Instructing follower 1c37142ee9c043d7a5aca01030e1bbf5 to start an election
I20260925 03:03:22.327548 24172 raft_consensus.cc:1067] T 78943162e8394d039f00afadd62893f3 P 5dae02dd94274f54ab9c52ab57a880b0 [term 1 LEADER]: Signalling peer 1c37142ee9c043d7a5aca01030e1bbf5 to start an election
I20260925 03:03:22.329021 24024 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "78943162e8394d039f00afadd62893f3"
dest_uuid: "1c37142ee9c043d7a5aca01030e1bbf5"
from {username='slave'} at 127.0.0.1:40122
I20260925 03:03:22.329592 24024 raft_consensus.cc:482] T 78943162e8394d039f00afadd62893f3 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 03:03:22.329934 24024 raft_consensus.cc:3037] T 78943162e8394d039f00afadd62893f3 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:03:22.332221 24149 consensus_queue.cc:1035] T 78943162e8394d039f00afadd62893f3 P 5dae02dd94274f54ab9c52ab57a880b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 03:03:22.338383 24024 raft_consensus.cc:504] T 78943162e8394d039f00afadd62893f3 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:22.339516 24023 raft_consensus.cc:1226] T 78943162e8394d039f00afadd62893f3 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 2 FOLLOWER]: Rejecting Update request from peer 5dae02dd94274f54ab9c52ab57a880b0 for earlier term 1. Current term is 2. Ops: []
I20260925 03:03:22.340827 24024 leader_election.cc:288] T 78943162e8394d039f00afadd62893f3 P 1c37142ee9c043d7a5aca01030e1bbf5 [CANDIDATE]: Term 2 election: Requested vote from peers e8496e8e22c447f9b270e862bbb272a3 (127.22.193.1:44975), 5dae02dd94274f54ab9c52ab57a880b0 (127.22.193.3:46383)
I20260925 03:03:22.341483 24097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78943162e8394d039f00afadd62893f3" candidate_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5dae02dd94274f54ab9c52ab57a880b0"
I20260925 03:03:22.342177 24097 raft_consensus.cc:3032] T 78943162e8394d039f00afadd62893f3 P 5dae02dd94274f54ab9c52ab57a880b0 [term 1 LEADER]: Stepping down as leader of term 1
I20260925 03:03:22.341483 23948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78943162e8394d039f00afadd62893f3" candidate_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e8496e8e22c447f9b270e862bbb272a3"
I20260925 03:03:22.342542 24097 raft_consensus.cc:729] T 78943162e8394d039f00afadd62893f3 P 5dae02dd94274f54ab9c52ab57a880b0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5dae02dd94274f54ab9c52ab57a880b0, State: Running, Role: LEADER
I20260925 03:03:22.343307 24097 consensus_queue.cc:260] T 78943162e8394d039f00afadd62893f3 P 5dae02dd94274f54ab9c52ab57a880b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:22.344147 24097 raft_consensus.cc:3037] T 78943162e8394d039f00afadd62893f3 P 5dae02dd94274f54ab9c52ab57a880b0 [term 1 FOLLOWER]: Advancing to term 2
W20260925 03:03:22.344780 24149 consensus_queue.cc:1159] T 78943162e8394d039f00afadd62893f3 P 5dae02dd94274f54ab9c52ab57a880b0 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" responder_term: 2 status { last_received { term: 1 index: 1 } last_committed_idx: 0 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 5dae02dd94274f54ab9c52ab57a880b0 for earlier term 1. Current term is 2. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260925 03:03:22.349094 23948 raft_consensus.cc:3037] T 78943162e8394d039f00afadd62893f3 P e8496e8e22c447f9b270e862bbb272a3 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:03:22.349292 24097 raft_consensus.cc:2454] T 78943162e8394d039f00afadd62893f3 P 5dae02dd94274f54ab9c52ab57a880b0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c37142ee9c043d7a5aca01030e1bbf5 in term 2.
I20260925 03:03:22.351303 23990 leader_election.cc:302] T 78943162e8394d039f00afadd62893f3 P 1c37142ee9c043d7a5aca01030e1bbf5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c37142ee9c043d7a5aca01030e1bbf5, 5dae02dd94274f54ab9c52ab57a880b0; no voters:
I20260925 03:03:22.352169 24152 raft_consensus.cc:2781] T 78943162e8394d039f00afadd62893f3 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 03:03:22.352584 24152 raft_consensus.cc:686] T 78943162e8394d039f00afadd62893f3 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 2 LEADER]: Becoming Leader. State: Replica: 1c37142ee9c043d7a5aca01030e1bbf5, State: Running, Role: LEADER
I20260925 03:03:22.353407 24152 consensus_queue.cc:237] T 78943162e8394d039f00afadd62893f3 P 1c37142ee9c043d7a5aca01030e1bbf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:22.355834 23948 raft_consensus.cc:2454] T 78943162e8394d039f00afadd62893f3 P e8496e8e22c447f9b270e862bbb272a3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c37142ee9c043d7a5aca01030e1bbf5 in term 2.
I20260925 03:03:22.362023 23846 catalog_manager.cc:5495] T 78943162e8394d039f00afadd62893f3 P 1c37142ee9c043d7a5aca01030e1bbf5 reported cstate change: term changed from 1 to 2, leader changed from 5dae02dd94274f54ab9c52ab57a880b0 (127.22.193.3) to 1c37142ee9c043d7a5aca01030e1bbf5 (127.22.193.2). New cstate: current_term: 2 leader_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } health_report { overall_health: UNKNOWN } } }
I20260925 03:03:22.368147 24149 consensus_queue.cc:1035] T f58a83ec6f4149108eb0567cd7817ab5 P 5dae02dd94274f54ab9c52ab57a880b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 03:03:22.378695 24190 consensus_queue.cc:1035] T f58a83ec6f4149108eb0567cd7817ab5 P 5dae02dd94274f54ab9c52ab57a880b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 03:03:22.386157 24149 raft_consensus.cc:979] T f58a83ec6f4149108eb0567cd7817ab5 P 5dae02dd94274f54ab9c52ab57a880b0: : Instructing follower e8496e8e22c447f9b270e862bbb272a3 to start an election
I20260925 03:03:22.386622 24190 raft_consensus.cc:1067] T f58a83ec6f4149108eb0567cd7817ab5 P 5dae02dd94274f54ab9c52ab57a880b0 [term 1 LEADER]: Signalling peer e8496e8e22c447f9b270e862bbb272a3 to start an election
I20260925 03:03:22.388428 23948 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f58a83ec6f4149108eb0567cd7817ab5"
dest_uuid: "e8496e8e22c447f9b270e862bbb272a3"
from {username='slave'} at 127.0.0.1:50460
I20260925 03:03:22.388996 23948 raft_consensus.cc:482] T f58a83ec6f4149108eb0567cd7817ab5 P e8496e8e22c447f9b270e862bbb272a3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 03:03:22.389339 23948 raft_consensus.cc:3037] T f58a83ec6f4149108eb0567cd7817ab5 P e8496e8e22c447f9b270e862bbb272a3 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:03:22.389925 24180 consensus_queue.cc:1035] T 5055d207a0994afbbbc25a2c39455f87 P e8496e8e22c447f9b270e862bbb272a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 03:03:22.395579 23948 raft_consensus.cc:504] T f58a83ec6f4149108eb0567cd7817ab5 P e8496e8e22c447f9b270e862bbb272a3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:22.399573 24098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f58a83ec6f4149108eb0567cd7817ab5" candidate_uuid: "e8496e8e22c447f9b270e862bbb272a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5dae02dd94274f54ab9c52ab57a880b0"
I20260925 03:03:22.400178 24098 raft_consensus.cc:3032] T f58a83ec6f4149108eb0567cd7817ab5 P 5dae02dd94274f54ab9c52ab57a880b0 [term 1 LEADER]: Stepping down as leader of term 1
I20260925 03:03:22.400501 24098 raft_consensus.cc:729] T f58a83ec6f4149108eb0567cd7817ab5 P 5dae02dd94274f54ab9c52ab57a880b0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5dae02dd94274f54ab9c52ab57a880b0, State: Running, Role: LEADER
I20260925 03:03:22.400874 24024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f58a83ec6f4149108eb0567cd7817ab5" candidate_uuid: "e8496e8e22c447f9b270e862bbb272a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1c37142ee9c043d7a5aca01030e1bbf5"
I20260925 03:03:22.401432 24024 raft_consensus.cc:3037] T f58a83ec6f4149108eb0567cd7817ab5 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:03:22.401274 24098 consensus_queue.cc:260] T f58a83ec6f4149108eb0567cd7817ab5 P 5dae02dd94274f54ab9c52ab57a880b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:22.402405 24098 raft_consensus.cc:3037] T f58a83ec6f4149108eb0567cd7817ab5 P 5dae02dd94274f54ab9c52ab57a880b0 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:03:22.402366 24224 consensus_queue.cc:1035] T 5055d207a0994afbbbc25a2c39455f87 P e8496e8e22c447f9b270e862bbb272a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260925 03:03:22.408236 24024 raft_consensus.cc:2454] T f58a83ec6f4149108eb0567cd7817ab5 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8496e8e22c447f9b270e862bbb272a3 in term 2.
I20260925 03:03:22.409313 24098 raft_consensus.cc:2454] T f58a83ec6f4149108eb0567cd7817ab5 P 5dae02dd94274f54ab9c52ab57a880b0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8496e8e22c447f9b270e862bbb272a3 in term 2.
I20260925 03:03:22.398980 23948 leader_election.cc:288] T f58a83ec6f4149108eb0567cd7817ab5 P e8496e8e22c447f9b270e862bbb272a3 [CANDIDATE]: Term 2 election: Requested vote from peers 1c37142ee9c043d7a5aca01030e1bbf5 (127.22.193.2:40513), 5dae02dd94274f54ab9c52ab57a880b0 (127.22.193.3:46383)
I20260925 03:03:22.411876 23913 leader_election.cc:302] T f58a83ec6f4149108eb0567cd7817ab5 P e8496e8e22c447f9b270e862bbb272a3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c37142ee9c043d7a5aca01030e1bbf5, e8496e8e22c447f9b270e862bbb272a3; no voters:
I20260925 03:03:22.412825 24224 raft_consensus.cc:2781] T f58a83ec6f4149108eb0567cd7817ab5 P e8496e8e22c447f9b270e862bbb272a3 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 03:03:22.413522 24224 raft_consensus.cc:686] T f58a83ec6f4149108eb0567cd7817ab5 P e8496e8e22c447f9b270e862bbb272a3 [term 2 LEADER]: Becoming Leader. State: Replica: e8496e8e22c447f9b270e862bbb272a3, State: Running, Role: LEADER
I20260925 03:03:22.414310 24224 consensus_queue.cc:237] T f58a83ec6f4149108eb0567cd7817ab5 P e8496e8e22c447f9b270e862bbb272a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:22.422065 24180 raft_consensus.cc:979] T 5055d207a0994afbbbc25a2c39455f87 P e8496e8e22c447f9b270e862bbb272a3: : Instructing follower 1c37142ee9c043d7a5aca01030e1bbf5 to start an election
I20260925 03:03:22.421753 23847 catalog_manager.cc:5495] T f58a83ec6f4149108eb0567cd7817ab5 P e8496e8e22c447f9b270e862bbb272a3 reported cstate change: term changed from 1 to 2, leader changed from 5dae02dd94274f54ab9c52ab57a880b0 (127.22.193.3) to e8496e8e22c447f9b270e862bbb272a3 (127.22.193.1). New cstate: current_term: 2 leader_uuid: "e8496e8e22c447f9b270e862bbb272a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } health_report { overall_health: UNKNOWN } } }
I20260925 03:03:22.422518 24224 raft_consensus.cc:1067] T 5055d207a0994afbbbc25a2c39455f87 P e8496e8e22c447f9b270e862bbb272a3 [term 1 LEADER]: Signalling peer 1c37142ee9c043d7a5aca01030e1bbf5 to start an election
I20260925 03:03:22.424163 24023 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5055d207a0994afbbbc25a2c39455f87"
dest_uuid: "1c37142ee9c043d7a5aca01030e1bbf5"
from {username='slave'} at 127.0.0.1:40132
I20260925 03:03:22.424613 24023 raft_consensus.cc:482] T 5055d207a0994afbbbc25a2c39455f87 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 03:03:22.424932 24023 raft_consensus.cc:3037] T 5055d207a0994afbbbc25a2c39455f87 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:03:22.430195 24023 raft_consensus.cc:504] T 5055d207a0994afbbbc25a2c39455f87 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:22.432214 24152 raft_consensus.cc:979] T 7f1e503d876c45fa853edeaf3b836180 P 1c37142ee9c043d7a5aca01030e1bbf5: : Instructing follower e8496e8e22c447f9b270e862bbb272a3 to start an election
I20260925 03:03:22.432399 24023 leader_election.cc:288] T 5055d207a0994afbbbc25a2c39455f87 P 1c37142ee9c043d7a5aca01030e1bbf5 [CANDIDATE]: Term 2 election: Requested vote from peers e8496e8e22c447f9b270e862bbb272a3 (127.22.193.1:44975), 5dae02dd94274f54ab9c52ab57a880b0 (127.22.193.3:46383)
I20260925 03:03:22.432654 24152 raft_consensus.cc:1067] T 7f1e503d876c45fa853edeaf3b836180 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 1 LEADER]: Signalling peer e8496e8e22c447f9b270e862bbb272a3 to start an election
I20260925 03:03:22.433238 23949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5055d207a0994afbbbc25a2c39455f87" candidate_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e8496e8e22c447f9b270e862bbb272a3"
I20260925 03:03:22.433797 23949 raft_consensus.cc:3032] T 5055d207a0994afbbbc25a2c39455f87 P e8496e8e22c447f9b270e862bbb272a3 [term 1 LEADER]: Stepping down as leader of term 1
I20260925 03:03:22.434100 23949 raft_consensus.cc:729] T 5055d207a0994afbbbc25a2c39455f87 P e8496e8e22c447f9b270e862bbb272a3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e8496e8e22c447f9b270e862bbb272a3, State: Running, Role: LEADER
I20260925 03:03:22.434784 23949 consensus_queue.cc:260] T 5055d207a0994afbbbc25a2c39455f87 P e8496e8e22c447f9b270e862bbb272a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:22.435882 23949 raft_consensus.cc:3037] T 5055d207a0994afbbbc25a2c39455f87 P e8496e8e22c447f9b270e862bbb272a3 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:03:22.437490 24098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5055d207a0994afbbbc25a2c39455f87" candidate_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5dae02dd94274f54ab9c52ab57a880b0"
I20260925 03:03:22.438062 24098 raft_consensus.cc:3037] T 5055d207a0994afbbbc25a2c39455f87 P 5dae02dd94274f54ab9c52ab57a880b0 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:03:22.438632 23948 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7f1e503d876c45fa853edeaf3b836180"
dest_uuid: "e8496e8e22c447f9b270e862bbb272a3"
from {username='slave'} at 127.0.0.1:50454
I20260925 03:03:22.439396 23948 raft_consensus.cc:482] T 7f1e503d876c45fa853edeaf3b836180 P e8496e8e22c447f9b270e862bbb272a3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 03:03:22.439787 23948 raft_consensus.cc:3037] T 7f1e503d876c45fa853edeaf3b836180 P e8496e8e22c447f9b270e862bbb272a3 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:03:22.445520 23948 raft_consensus.cc:504] T 7f1e503d876c45fa853edeaf3b836180 P e8496e8e22c447f9b270e862bbb272a3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:22.445627 24098 raft_consensus.cc:2454] T 5055d207a0994afbbbc25a2c39455f87 P 5dae02dd94274f54ab9c52ab57a880b0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c37142ee9c043d7a5aca01030e1bbf5 in term 2.
I20260925 03:03:22.445972 23949 raft_consensus.cc:2454] T 5055d207a0994afbbbc25a2c39455f87 P e8496e8e22c447f9b270e862bbb272a3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c37142ee9c043d7a5aca01030e1bbf5 in term 2.
I20260925 03:03:22.447098 23988 leader_election.cc:302] T 5055d207a0994afbbbc25a2c39455f87 P 1c37142ee9c043d7a5aca01030e1bbf5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c37142ee9c043d7a5aca01030e1bbf5, e8496e8e22c447f9b270e862bbb272a3; no voters:
I20260925 03:03:22.447335 23948 leader_election.cc:288] T 7f1e503d876c45fa853edeaf3b836180 P e8496e8e22c447f9b270e862bbb272a3 [CANDIDATE]: Term 2 election: Requested vote from peers 1c37142ee9c043d7a5aca01030e1bbf5 (127.22.193.2:40513), 5dae02dd94274f54ab9c52ab57a880b0 (127.22.193.3:46383)
I20260925 03:03:22.448165 24154 raft_consensus.cc:2781] T 5055d207a0994afbbbc25a2c39455f87 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 03:03:22.448431 24098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f1e503d876c45fa853edeaf3b836180" candidate_uuid: "e8496e8e22c447f9b270e862bbb272a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5dae02dd94274f54ab9c52ab57a880b0"
I20260925 03:03:22.448628 24154 raft_consensus.cc:686] T 5055d207a0994afbbbc25a2c39455f87 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 2 LEADER]: Becoming Leader. State: Replica: 1c37142ee9c043d7a5aca01030e1bbf5, State: Running, Role: LEADER
I20260925 03:03:22.449054 24023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f1e503d876c45fa853edeaf3b836180" candidate_uuid: "e8496e8e22c447f9b270e862bbb272a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1c37142ee9c043d7a5aca01030e1bbf5"
I20260925 03:03:22.449640 24023 raft_consensus.cc:3032] T 7f1e503d876c45fa853edeaf3b836180 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 1 LEADER]: Stepping down as leader of term 1
I20260925 03:03:22.450016 24023 raft_consensus.cc:729] T 7f1e503d876c45fa853edeaf3b836180 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1c37142ee9c043d7a5aca01030e1bbf5, State: Running, Role: LEADER
I20260925 03:03:22.449559 24154 consensus_queue.cc:237] T 5055d207a0994afbbbc25a2c39455f87 P 1c37142ee9c043d7a5aca01030e1bbf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:22.450762 24023 consensus_queue.cc:260] T 7f1e503d876c45fa853edeaf3b836180 P 1c37142ee9c043d7a5aca01030e1bbf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:22.451984 24023 raft_consensus.cc:3037] T 7f1e503d876c45fa853edeaf3b836180 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:03:22.449035 24098 raft_consensus.cc:3037] T 7f1e503d876c45fa853edeaf3b836180 P 5dae02dd94274f54ab9c52ab57a880b0 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:03:22.454257 24152 raft_consensus.cc:979] T c9f3dfa77c7845a49f80417befebb978 P 1c37142ee9c043d7a5aca01030e1bbf5: : Instructing follower e8496e8e22c447f9b270e862bbb272a3 to start an election
I20260925 03:03:22.454767 24154 raft_consensus.cc:1067] T c9f3dfa77c7845a49f80417befebb978 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 1 LEADER]: Signalling peer e8496e8e22c447f9b270e862bbb272a3 to start an election
I20260925 03:03:22.456224 23948 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c9f3dfa77c7845a49f80417befebb978"
dest_uuid: "e8496e8e22c447f9b270e862bbb272a3"
from {username='slave'} at 127.0.0.1:50454
I20260925 03:03:22.456809 23948 raft_consensus.cc:482] T c9f3dfa77c7845a49f80417befebb978 P e8496e8e22c447f9b270e862bbb272a3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 03:03:22.457288 23948 raft_consensus.cc:3037] T c9f3dfa77c7845a49f80417befebb978 P e8496e8e22c447f9b270e862bbb272a3 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:03:22.458139 23847 catalog_manager.cc:5495] T 5055d207a0994afbbbc25a2c39455f87 P 1c37142ee9c043d7a5aca01030e1bbf5 reported cstate change: term changed from 1 to 2, leader changed from e8496e8e22c447f9b270e862bbb272a3 (127.22.193.1) to 1c37142ee9c043d7a5aca01030e1bbf5 (127.22.193.2). New cstate: current_term: 2 leader_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } health_report { overall_health: UNKNOWN } } }
I20260925 03:03:22.463332 23948 raft_consensus.cc:504] T c9f3dfa77c7845a49f80417befebb978 P e8496e8e22c447f9b270e862bbb272a3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:22.466003 23948 leader_election.cc:288] T c9f3dfa77c7845a49f80417befebb978 P e8496e8e22c447f9b270e862bbb272a3 [CANDIDATE]: Term 2 election: Requested vote from peers 1c37142ee9c043d7a5aca01030e1bbf5 (127.22.193.2:40513), 5dae02dd94274f54ab9c52ab57a880b0 (127.22.193.3:46383)
I20260925 03:03:22.466225 24097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9f3dfa77c7845a49f80417befebb978" candidate_uuid: "e8496e8e22c447f9b270e862bbb272a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5dae02dd94274f54ab9c52ab57a880b0"
I20260925 03:03:22.466805 24097 raft_consensus.cc:3037] T c9f3dfa77c7845a49f80417befebb978 P 5dae02dd94274f54ab9c52ab57a880b0 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:03:22.467353 24024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9f3dfa77c7845a49f80417befebb978" candidate_uuid: "e8496e8e22c447f9b270e862bbb272a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1c37142ee9c043d7a5aca01030e1bbf5"
I20260925 03:03:22.467952 24024 raft_consensus.cc:3032] T c9f3dfa77c7845a49f80417befebb978 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 1 LEADER]: Stepping down as leader of term 1
I20260925 03:03:22.468266 24024 raft_consensus.cc:729] T c9f3dfa77c7845a49f80417befebb978 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1c37142ee9c043d7a5aca01030e1bbf5, State: Running, Role: LEADER
I20260925 03:03:22.469280 24098 raft_consensus.cc:2454] T 7f1e503d876c45fa853edeaf3b836180 P 5dae02dd94274f54ab9c52ab57a880b0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8496e8e22c447f9b270e862bbb272a3 in term 2.
I20260925 03:03:22.469029 24024 consensus_queue.cc:260] T c9f3dfa77c7845a49f80417befebb978 P 1c37142ee9c043d7a5aca01030e1bbf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:22.470301 24024 raft_consensus.cc:3037] T c9f3dfa77c7845a49f80417befebb978 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:03:22.470345 23915 leader_election.cc:302] T 7f1e503d876c45fa853edeaf3b836180 P e8496e8e22c447f9b270e862bbb272a3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dae02dd94274f54ab9c52ab57a880b0, e8496e8e22c447f9b270e862bbb272a3; no voters:
I20260925 03:03:22.471189 24224 raft_consensus.cc:2781] T 7f1e503d876c45fa853edeaf3b836180 P e8496e8e22c447f9b270e862bbb272a3 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 03:03:22.471626 24224 raft_consensus.cc:686] T 7f1e503d876c45fa853edeaf3b836180 P e8496e8e22c447f9b270e862bbb272a3 [term 2 LEADER]: Becoming Leader. State: Replica: e8496e8e22c447f9b270e862bbb272a3, State: Running, Role: LEADER
I20260925 03:03:22.472020 24023 raft_consensus.cc:2454] T 7f1e503d876c45fa853edeaf3b836180 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8496e8e22c447f9b270e862bbb272a3 in term 2.
I20260925 03:03:22.472401 24224 consensus_queue.cc:237] T 7f1e503d876c45fa853edeaf3b836180 P e8496e8e22c447f9b270e862bbb272a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:22.473268 24097 raft_consensus.cc:2454] T c9f3dfa77c7845a49f80417befebb978 P 5dae02dd94274f54ab9c52ab57a880b0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8496e8e22c447f9b270e862bbb272a3 in term 2.
I20260925 03:03:22.474366 23915 leader_election.cc:302] T c9f3dfa77c7845a49f80417befebb978 P e8496e8e22c447f9b270e862bbb272a3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dae02dd94274f54ab9c52ab57a880b0, e8496e8e22c447f9b270e862bbb272a3; no voters:
I20260925 03:03:22.475533 24180 raft_consensus.cc:2781] T c9f3dfa77c7845a49f80417befebb978 P e8496e8e22c447f9b270e862bbb272a3 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 03:03:22.476413 24024 raft_consensus.cc:2454] T c9f3dfa77c7845a49f80417befebb978 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8496e8e22c447f9b270e862bbb272a3 in term 2.
I20260925 03:03:22.478641 24180 raft_consensus.cc:686] T c9f3dfa77c7845a49f80417befebb978 P e8496e8e22c447f9b270e862bbb272a3 [term 2 LEADER]: Becoming Leader. State: Replica: e8496e8e22c447f9b270e862bbb272a3, State: Running, Role: LEADER
I20260925 03:03:22.479411 24180 consensus_queue.cc:237] T c9f3dfa77c7845a49f80417befebb978 P e8496e8e22c447f9b270e862bbb272a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:22.480834 23847 catalog_manager.cc:5495] T 7f1e503d876c45fa853edeaf3b836180 P e8496e8e22c447f9b270e862bbb272a3 reported cstate change: term changed from 1 to 2, leader changed from 1c37142ee9c043d7a5aca01030e1bbf5 (127.22.193.2) to e8496e8e22c447f9b270e862bbb272a3 (127.22.193.1). New cstate: current_term: 2 leader_uuid: "e8496e8e22c447f9b270e862bbb272a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } health_report { overall_health: UNKNOWN } } }
I20260925 03:03:22.485870 24170 consensus_queue.cc:1035] T 48c4fddba997473f8a501507277b8860 P 5dae02dd94274f54ab9c52ab57a880b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 03:03:22.492875 24149 raft_consensus.cc:979] T 48c4fddba997473f8a501507277b8860 P 5dae02dd94274f54ab9c52ab57a880b0: : Instructing follower e8496e8e22c447f9b270e862bbb272a3 to start an election
I20260925 03:03:22.493324 24172 raft_consensus.cc:1067] T 48c4fddba997473f8a501507277b8860 P 5dae02dd94274f54ab9c52ab57a880b0 [term 1 LEADER]: Signalling peer e8496e8e22c447f9b270e862bbb272a3 to start an election
I20260925 03:03:22.494839 23948 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "48c4fddba997473f8a501507277b8860"
dest_uuid: "e8496e8e22c447f9b270e862bbb272a3"
from {username='slave'} at 127.0.0.1:50460
I20260925 03:03:22.494764 23847 catalog_manager.cc:5495] T c9f3dfa77c7845a49f80417befebb978 P e8496e8e22c447f9b270e862bbb272a3 reported cstate change: term changed from 1 to 2, leader changed from 1c37142ee9c043d7a5aca01030e1bbf5 (127.22.193.2) to e8496e8e22c447f9b270e862bbb272a3 (127.22.193.1). New cstate: current_term: 2 leader_uuid: "e8496e8e22c447f9b270e862bbb272a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } health_report { overall_health: UNKNOWN } } }
I20260925 03:03:22.495749 23948 raft_consensus.cc:482] T 48c4fddba997473f8a501507277b8860 P e8496e8e22c447f9b270e862bbb272a3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 03:03:22.496152 23948 raft_consensus.cc:3037] T 48c4fddba997473f8a501507277b8860 P e8496e8e22c447f9b270e862bbb272a3 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:03:22.500967 24172 consensus_queue.cc:1035] T 48c4fddba997473f8a501507277b8860 P 5dae02dd94274f54ab9c52ab57a880b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 03:03:22.502878 23948 raft_consensus.cc:504] T 48c4fddba997473f8a501507277b8860 P e8496e8e22c447f9b270e862bbb272a3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:22.504480 23948 leader_election.cc:288] T 48c4fddba997473f8a501507277b8860 P e8496e8e22c447f9b270e862bbb272a3 [CANDIDATE]: Term 2 election: Requested vote from peers 1c37142ee9c043d7a5aca01030e1bbf5 (127.22.193.2:40513), 5dae02dd94274f54ab9c52ab57a880b0 (127.22.193.3:46383)
I20260925 03:03:22.505285 24023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48c4fddba997473f8a501507277b8860" candidate_uuid: "e8496e8e22c447f9b270e862bbb272a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1c37142ee9c043d7a5aca01030e1bbf5"
I20260925 03:03:22.505841 24023 raft_consensus.cc:3037] T 48c4fddba997473f8a501507277b8860 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:03:22.505995 24097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48c4fddba997473f8a501507277b8860" candidate_uuid: "e8496e8e22c447f9b270e862bbb272a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5dae02dd94274f54ab9c52ab57a880b0"
I20260925 03:03:22.506513 24097 raft_consensus.cc:3032] T 48c4fddba997473f8a501507277b8860 P 5dae02dd94274f54ab9c52ab57a880b0 [term 1 LEADER]: Stepping down as leader of term 1
I20260925 03:03:22.506740 24097 raft_consensus.cc:729] T 48c4fddba997473f8a501507277b8860 P 5dae02dd94274f54ab9c52ab57a880b0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5dae02dd94274f54ab9c52ab57a880b0, State: Running, Role: LEADER
I20260925 03:03:22.507385 24097 consensus_queue.cc:260] T 48c4fddba997473f8a501507277b8860 P 5dae02dd94274f54ab9c52ab57a880b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:22.508178 24097 raft_consensus.cc:3037] T 48c4fddba997473f8a501507277b8860 P 5dae02dd94274f54ab9c52ab57a880b0 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:03:22.511811 24023 raft_consensus.cc:2454] T 48c4fddba997473f8a501507277b8860 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8496e8e22c447f9b270e862bbb272a3 in term 2.
I20260925 03:03:22.512609 24024 raft_consensus.cc:1226] T 48c4fddba997473f8a501507277b8860 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 2 FOLLOWER]: Rejecting Update request from peer 5dae02dd94274f54ab9c52ab57a880b0 for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20260925 03:03:22.513116 23913 leader_election.cc:302] T 48c4fddba997473f8a501507277b8860 P e8496e8e22c447f9b270e862bbb272a3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c37142ee9c043d7a5aca01030e1bbf5, e8496e8e22c447f9b270e862bbb272a3; no voters:
I20260925 03:03:22.513955 24224 raft_consensus.cc:2781] T 48c4fddba997473f8a501507277b8860 P e8496e8e22c447f9b270e862bbb272a3 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 03:03:22.514642 24224 raft_consensus.cc:686] T 48c4fddba997473f8a501507277b8860 P e8496e8e22c447f9b270e862bbb272a3 [term 2 LEADER]: Becoming Leader. State: Replica: e8496e8e22c447f9b270e862bbb272a3, State: Running, Role: LEADER
I20260925 03:03:22.513847 24097 raft_consensus.cc:2454] T 48c4fddba997473f8a501507277b8860 P 5dae02dd94274f54ab9c52ab57a880b0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8496e8e22c447f9b270e862bbb272a3 in term 2.
I20260925 03:03:22.515461 24224 consensus_queue.cc:237] T 48c4fddba997473f8a501507277b8860 P e8496e8e22c447f9b270e862bbb272a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:22.521711 23847 catalog_manager.cc:5495] T 48c4fddba997473f8a501507277b8860 P e8496e8e22c447f9b270e862bbb272a3 reported cstate change: term changed from 1 to 2, leader changed from 5dae02dd94274f54ab9c52ab57a880b0 (127.22.193.3) to e8496e8e22c447f9b270e862bbb272a3 (127.22.193.1). New cstate: current_term: 2 leader_uuid: "e8496e8e22c447f9b270e862bbb272a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } health_report { overall_health: UNKNOWN } } }
I20260925 03:03:22.802050 24170 raft_consensus.cc:979] T 845902c057d54c5ab9ddc861f622125c P 5dae02dd94274f54ab9c52ab57a880b0: : Instructing follower e8496e8e22c447f9b270e862bbb272a3 to start an election
I20260925 03:03:22.802546 24172 raft_consensus.cc:1067] T 845902c057d54c5ab9ddc861f622125c P 5dae02dd94274f54ab9c52ab57a880b0 [term 1 LEADER]: Signalling peer e8496e8e22c447f9b270e862bbb272a3 to start an election
I20260925 03:03:22.804054 23948 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "845902c057d54c5ab9ddc861f622125c"
dest_uuid: "e8496e8e22c447f9b270e862bbb272a3"
from {username='slave'} at 127.0.0.1:50460
I20260925 03:03:22.804535 23948 raft_consensus.cc:482] T 845902c057d54c5ab9ddc861f622125c P e8496e8e22c447f9b270e862bbb272a3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 03:03:22.804875 23948 raft_consensus.cc:3037] T 845902c057d54c5ab9ddc861f622125c P e8496e8e22c447f9b270e862bbb272a3 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:03:22.810050 23948 raft_consensus.cc:504] T 845902c057d54c5ab9ddc861f622125c P e8496e8e22c447f9b270e862bbb272a3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:22.811630 23948 leader_election.cc:288] T 845902c057d54c5ab9ddc861f622125c P e8496e8e22c447f9b270e862bbb272a3 [CANDIDATE]: Term 2 election: Requested vote from peers 1c37142ee9c043d7a5aca01030e1bbf5 (127.22.193.2:40513), 5dae02dd94274f54ab9c52ab57a880b0 (127.22.193.3:46383)
I20260925 03:03:22.812638 24024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "845902c057d54c5ab9ddc861f622125c" candidate_uuid: "e8496e8e22c447f9b270e862bbb272a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1c37142ee9c043d7a5aca01030e1bbf5"
I20260925 03:03:22.812817 24097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "845902c057d54c5ab9ddc861f622125c" candidate_uuid: "e8496e8e22c447f9b270e862bbb272a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5dae02dd94274f54ab9c52ab57a880b0"
I20260925 03:03:22.813230 24024 raft_consensus.cc:3037] T 845902c057d54c5ab9ddc861f622125c P 1c37142ee9c043d7a5aca01030e1bbf5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:03:22.813344 24097 raft_consensus.cc:3032] T 845902c057d54c5ab9ddc861f622125c P 5dae02dd94274f54ab9c52ab57a880b0 [term 1 LEADER]: Stepping down as leader of term 1
I20260925 03:03:22.813679 24097 raft_consensus.cc:729] T 845902c057d54c5ab9ddc861f622125c P 5dae02dd94274f54ab9c52ab57a880b0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5dae02dd94274f54ab9c52ab57a880b0, State: Running, Role: LEADER
I20260925 03:03:22.814409 24097 consensus_queue.cc:260] T 845902c057d54c5ab9ddc861f622125c P 5dae02dd94274f54ab9c52ab57a880b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:22.815563 24097 raft_consensus.cc:3037] T 845902c057d54c5ab9ddc861f622125c P 5dae02dd94274f54ab9c52ab57a880b0 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:03:22.818585 24024 raft_consensus.cc:2454] T 845902c057d54c5ab9ddc861f622125c P 1c37142ee9c043d7a5aca01030e1bbf5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8496e8e22c447f9b270e862bbb272a3 in term 2.
I20260925 03:03:22.819613 23913 leader_election.cc:302] T 845902c057d54c5ab9ddc861f622125c P e8496e8e22c447f9b270e862bbb272a3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c37142ee9c043d7a5aca01030e1bbf5, e8496e8e22c447f9b270e862bbb272a3; no voters:
I20260925 03:03:22.820514 24224 raft_consensus.cc:2781] T 845902c057d54c5ab9ddc861f622125c P e8496e8e22c447f9b270e862bbb272a3 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 03:03:22.821165 24097 raft_consensus.cc:2454] T 845902c057d54c5ab9ddc861f622125c P 5dae02dd94274f54ab9c52ab57a880b0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8496e8e22c447f9b270e862bbb272a3 in term 2.
I20260925 03:03:22.821185 24224 raft_consensus.cc:686] T 845902c057d54c5ab9ddc861f622125c P e8496e8e22c447f9b270e862bbb272a3 [term 2 LEADER]: Becoming Leader. State: Replica: e8496e8e22c447f9b270e862bbb272a3, State: Running, Role: LEADER
I20260925 03:03:22.822134 24224 consensus_queue.cc:237] T 845902c057d54c5ab9ddc861f622125c P e8496e8e22c447f9b270e862bbb272a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:22.823652 24097 raft_consensus.cc:1261] T 78943162e8394d039f00afadd62893f3 P 5dae02dd94274f54ab9c52ab57a880b0 [term 2 FOLLOWER]: Refusing update from remote peer 1c37142ee9c043d7a5aca01030e1bbf5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260925 03:03:22.824966 24154 consensus_queue.cc:1035] T 78943162e8394d039f00afadd62893f3 P 1c37142ee9c043d7a5aca01030e1bbf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260925 03:03:22.830334 23847 catalog_manager.cc:5495] T 845902c057d54c5ab9ddc861f622125c P e8496e8e22c447f9b270e862bbb272a3 reported cstate change: term changed from 1 to 2, leader changed from 5dae02dd94274f54ab9c52ab57a880b0 (127.22.193.3) to e8496e8e22c447f9b270e862bbb272a3 (127.22.193.1). New cstate: current_term: 2 leader_uuid: "e8496e8e22c447f9b270e862bbb272a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } health_report { overall_health: UNKNOWN } } }
I20260925 03:03:22.838438 23948 raft_consensus.cc:1261] T 78943162e8394d039f00afadd62893f3 P e8496e8e22c447f9b270e862bbb272a3 [term 2 FOLLOWER]: Refusing update from remote peer 1c37142ee9c043d7a5aca01030e1bbf5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260925 03:03:22.840111 24164 consensus_queue.cc:1035] T 78943162e8394d039f00afadd62893f3 P 1c37142ee9c043d7a5aca01030e1bbf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260925 03:03:22.954486 24097 raft_consensus.cc:1261] T f58a83ec6f4149108eb0567cd7817ab5 P 5dae02dd94274f54ab9c52ab57a880b0 [term 2 FOLLOWER]: Refusing update from remote peer e8496e8e22c447f9b270e862bbb272a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260925 03:03:22.955871 24224 consensus_queue.cc:1035] T f58a83ec6f4149108eb0567cd7817ab5 P e8496e8e22c447f9b270e862bbb272a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260925 03:03:22.964192 24024 raft_consensus.cc:1261] T f58a83ec6f4149108eb0567cd7817ab5 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 2 FOLLOWER]: Refusing update from remote peer e8496e8e22c447f9b270e862bbb272a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260925 03:03:22.965693 24180 consensus_queue.cc:1035] T f58a83ec6f4149108eb0567cd7817ab5 P e8496e8e22c447f9b270e862bbb272a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260925 03:03:22.980803 24024 raft_consensus.cc:1261] T 7f1e503d876c45fa853edeaf3b836180 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 2 FOLLOWER]: Refusing update from remote peer e8496e8e22c447f9b270e862bbb272a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260925 03:03:22.981990 24180 consensus_queue.cc:1035] T 7f1e503d876c45fa853edeaf3b836180 P e8496e8e22c447f9b270e862bbb272a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260925 03:03:22.992430 24097 raft_consensus.cc:1261] T 7f1e503d876c45fa853edeaf3b836180 P 5dae02dd94274f54ab9c52ab57a880b0 [term 2 FOLLOWER]: Refusing update from remote peer e8496e8e22c447f9b270e862bbb272a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260925 03:03:22.995247 24180 consensus_queue.cc:1035] T 7f1e503d876c45fa853edeaf3b836180 P e8496e8e22c447f9b270e862bbb272a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260925 03:03:23.002456 24097 raft_consensus.cc:1261] T c9f3dfa77c7845a49f80417befebb978 P 5dae02dd94274f54ab9c52ab57a880b0 [term 2 FOLLOWER]: Refusing update from remote peer e8496e8e22c447f9b270e862bbb272a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260925 03:03:23.003772 24180 consensus_queue.cc:1035] T c9f3dfa77c7845a49f80417befebb978 P e8496e8e22c447f9b270e862bbb272a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260925 03:03:23.009757 23948 raft_consensus.cc:1261] T 5055d207a0994afbbbc25a2c39455f87 P e8496e8e22c447f9b270e862bbb272a3 [term 2 FOLLOWER]: Refusing update from remote peer 1c37142ee9c043d7a5aca01030e1bbf5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260925 03:03:23.011339 24164 consensus_queue.cc:1035] T 5055d207a0994afbbbc25a2c39455f87 P 1c37142ee9c043d7a5aca01030e1bbf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260925 03:03:23.012991 24024 raft_consensus.cc:1261] T c9f3dfa77c7845a49f80417befebb978 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 2 FOLLOWER]: Refusing update from remote peer e8496e8e22c447f9b270e862bbb272a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260925 03:03:23.015069 24226 consensus_queue.cc:1035] T c9f3dfa77c7845a49f80417befebb978 P e8496e8e22c447f9b270e862bbb272a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260925 03:03:23.016508 24097 raft_consensus.cc:1261] T 48c4fddba997473f8a501507277b8860 P 5dae02dd94274f54ab9c52ab57a880b0 [term 2 FOLLOWER]: Refusing update from remote peer e8496e8e22c447f9b270e862bbb272a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260925 03:03:23.018803 24226 consensus_queue.cc:1035] T 48c4fddba997473f8a501507277b8860 P e8496e8e22c447f9b270e862bbb272a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260925 03:03:23.026700 24098 raft_consensus.cc:1261] T 5055d207a0994afbbbc25a2c39455f87 P 5dae02dd94274f54ab9c52ab57a880b0 [term 2 FOLLOWER]: Refusing update from remote peer 1c37142ee9c043d7a5aca01030e1bbf5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260925 03:03:23.028357 24164 consensus_queue.cc:1035] T 5055d207a0994afbbbc25a2c39455f87 P 1c37142ee9c043d7a5aca01030e1bbf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260925 03:03:23.031564 24024 raft_consensus.cc:1261] T 48c4fddba997473f8a501507277b8860 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 2 FOLLOWER]: Refusing update from remote peer e8496e8e22c447f9b270e862bbb272a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260925 03:03:23.034252 24180 consensus_queue.cc:1035] T 48c4fddba997473f8a501507277b8860 P e8496e8e22c447f9b270e862bbb272a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 03:03:23.269013 24098 raft_consensus.cc:1261] T 845902c057d54c5ab9ddc861f622125c P 5dae02dd94274f54ab9c52ab57a880b0 [term 2 FOLLOWER]: Refusing update from remote peer e8496e8e22c447f9b270e862bbb272a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260925 03:03:23.270192 24180 consensus_queue.cc:1035] T 845902c057d54c5ab9ddc861f622125c P e8496e8e22c447f9b270e862bbb272a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260925 03:03:23.278918 24024 raft_consensus.cc:1261] T 845902c057d54c5ab9ddc861f622125c P 1c37142ee9c043d7a5aca01030e1bbf5 [term 2 FOLLOWER]: Refusing update from remote peer e8496e8e22c447f9b270e862bbb272a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260925 03:03:23.280789 24229 consensus_queue.cc:1035] T 845902c057d54c5ab9ddc861f622125c P e8496e8e22c447f9b270e862bbb272a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260925 03:03:25.329272 23300 auto_leader_rebalancer-test.cc:339] The leader distribution is
I20260925 03:03:25.329921 23300 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260925 03:03:25.348096 23949 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "845902c057d54c5ab9ddc861f622125c"
dest_uuid: "e8496e8e22c447f9b270e862bbb272a3"
mode: GRACEFUL
new_leader_uuid: "5dae02dd94274f54ab9c52ab57a880b0"
from {username='slave'} at 127.0.0.1:57306
I20260925 03:03:25.348635 23949 raft_consensus.cc:595] T 845902c057d54c5ab9ddc861f622125c P e8496e8e22c447f9b270e862bbb272a3 [term 2 LEADER]: Received request to transfer leadership to TS 5dae02dd94274f54ab9c52ab57a880b0
I20260925 03:03:25.351033 23949 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f58a83ec6f4149108eb0567cd7817ab5"
dest_uuid: "e8496e8e22c447f9b270e862bbb272a3"
mode: GRACEFUL
new_leader_uuid: "5dae02dd94274f54ab9c52ab57a880b0"
from {username='slave'} at 127.0.0.1:57306
I20260925 03:03:25.351482 23949 raft_consensus.cc:595] T f58a83ec6f4149108eb0567cd7817ab5 P e8496e8e22c447f9b270e862bbb272a3 [term 2 LEADER]: Received request to transfer leadership to TS 5dae02dd94274f54ab9c52ab57a880b0
I20260925 03:03:25.352310 23300 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260925 03:03:25.352764 23300 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260925 03:03:25.559664 24236 raft_consensus.cc:979] T f58a83ec6f4149108eb0567cd7817ab5 P e8496e8e22c447f9b270e862bbb272a3: : Instructing follower 5dae02dd94274f54ab9c52ab57a880b0 to start an election
I20260925 03:03:25.560111 24224 raft_consensus.cc:1067] T f58a83ec6f4149108eb0567cd7817ab5 P e8496e8e22c447f9b270e862bbb272a3 [term 2 LEADER]: Signalling peer 5dae02dd94274f54ab9c52ab57a880b0 to start an election
I20260925 03:03:25.561558 24098 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f58a83ec6f4149108eb0567cd7817ab5"
dest_uuid: "5dae02dd94274f54ab9c52ab57a880b0"
from {username='slave'} at 127.0.0.1:46490
I20260925 03:03:25.562158 24098 raft_consensus.cc:482] T f58a83ec6f4149108eb0567cd7817ab5 P 5dae02dd94274f54ab9c52ab57a880b0 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 03:03:25.562484 24098 raft_consensus.cc:3037] T f58a83ec6f4149108eb0567cd7817ab5 P 5dae02dd94274f54ab9c52ab57a880b0 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:03:25.567544 24098 raft_consensus.cc:504] T f58a83ec6f4149108eb0567cd7817ab5 P 5dae02dd94274f54ab9c52ab57a880b0 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:25.569185 24098 leader_election.cc:288] T f58a83ec6f4149108eb0567cd7817ab5 P 5dae02dd94274f54ab9c52ab57a880b0 [CANDIDATE]: Term 3 election: Requested vote from peers 1c37142ee9c043d7a5aca01030e1bbf5 (127.22.193.2:40513), e8496e8e22c447f9b270e862bbb272a3 (127.22.193.1:44975)
I20260925 03:03:25.569933 24024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f58a83ec6f4149108eb0567cd7817ab5" candidate_uuid: "5dae02dd94274f54ab9c52ab57a880b0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "1c37142ee9c043d7a5aca01030e1bbf5"
I20260925 03:03:25.570564 24024 raft_consensus.cc:3037] T f58a83ec6f4149108eb0567cd7817ab5 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:03:25.570317 23949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f58a83ec6f4149108eb0567cd7817ab5" candidate_uuid: "5dae02dd94274f54ab9c52ab57a880b0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e8496e8e22c447f9b270e862bbb272a3"
I20260925 03:03:25.571022 23949 raft_consensus.cc:3032] T f58a83ec6f4149108eb0567cd7817ab5 P e8496e8e22c447f9b270e862bbb272a3 [term 2 LEADER]: Stepping down as leader of term 2
I20260925 03:03:25.571321 23949 raft_consensus.cc:729] T f58a83ec6f4149108eb0567cd7817ab5 P e8496e8e22c447f9b270e862bbb272a3 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e8496e8e22c447f9b270e862bbb272a3, State: Running, Role: LEADER
I20260925 03:03:25.572139 23949 consensus_queue.cc:260] T f58a83ec6f4149108eb0567cd7817ab5 P e8496e8e22c447f9b270e862bbb272a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:25.573380 23949 raft_consensus.cc:3037] T f58a83ec6f4149108eb0567cd7817ab5 P e8496e8e22c447f9b270e862bbb272a3 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:03:25.577243 24024 raft_consensus.cc:2454] T f58a83ec6f4149108eb0567cd7817ab5 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dae02dd94274f54ab9c52ab57a880b0 in term 3.
I20260925 03:03:25.578493 24062 leader_election.cc:302] T f58a83ec6f4149108eb0567cd7817ab5 P 5dae02dd94274f54ab9c52ab57a880b0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c37142ee9c043d7a5aca01030e1bbf5, 5dae02dd94274f54ab9c52ab57a880b0; no voters:
I20260925 03:03:25.579270 24172 raft_consensus.cc:2781] T f58a83ec6f4149108eb0567cd7817ab5 P 5dae02dd94274f54ab9c52ab57a880b0 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 03:03:25.579730 23949 raft_consensus.cc:2454] T f58a83ec6f4149108eb0567cd7817ab5 P e8496e8e22c447f9b270e862bbb272a3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dae02dd94274f54ab9c52ab57a880b0 in term 3.
I20260925 03:03:25.580224 24172 raft_consensus.cc:686] T f58a83ec6f4149108eb0567cd7817ab5 P 5dae02dd94274f54ab9c52ab57a880b0 [term 3 LEADER]: Becoming Leader. State: Replica: 5dae02dd94274f54ab9c52ab57a880b0, State: Running, Role: LEADER
I20260925 03:03:25.581007 24172 consensus_queue.cc:237] T f58a83ec6f4149108eb0567cd7817ab5 P 5dae02dd94274f54ab9c52ab57a880b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:25.588011 23847 catalog_manager.cc:5495] T f58a83ec6f4149108eb0567cd7817ab5 P 5dae02dd94274f54ab9c52ab57a880b0 reported cstate change: term changed from 2 to 3, leader changed from e8496e8e22c447f9b270e862bbb272a3 (127.22.193.1) to 5dae02dd94274f54ab9c52ab57a880b0 (127.22.193.3). New cstate: current_term: 3 leader_uuid: "5dae02dd94274f54ab9c52ab57a880b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } health_report { overall_health: HEALTHY } } }
I20260925 03:03:25.637439 24236 raft_consensus.cc:979] T 845902c057d54c5ab9ddc861f622125c P e8496e8e22c447f9b270e862bbb272a3: : Instructing follower 5dae02dd94274f54ab9c52ab57a880b0 to start an election
I20260925 03:03:25.637845 24224 raft_consensus.cc:1067] T 845902c057d54c5ab9ddc861f622125c P e8496e8e22c447f9b270e862bbb272a3 [term 2 LEADER]: Signalling peer 5dae02dd94274f54ab9c52ab57a880b0 to start an election
I20260925 03:03:25.639225 24098 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "845902c057d54c5ab9ddc861f622125c"
dest_uuid: "5dae02dd94274f54ab9c52ab57a880b0"
from {username='slave'} at 127.0.0.1:46490
I20260925 03:03:25.639729 24098 raft_consensus.cc:482] T 845902c057d54c5ab9ddc861f622125c P 5dae02dd94274f54ab9c52ab57a880b0 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 03:03:25.640008 24098 raft_consensus.cc:3037] T 845902c057d54c5ab9ddc861f622125c P 5dae02dd94274f54ab9c52ab57a880b0 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:03:25.645605 24098 raft_consensus.cc:504] T 845902c057d54c5ab9ddc861f622125c P 5dae02dd94274f54ab9c52ab57a880b0 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:25.647346 24098 leader_election.cc:288] T 845902c057d54c5ab9ddc861f622125c P 5dae02dd94274f54ab9c52ab57a880b0 [CANDIDATE]: Term 3 election: Requested vote from peers e8496e8e22c447f9b270e862bbb272a3 (127.22.193.1:44975), 1c37142ee9c043d7a5aca01030e1bbf5 (127.22.193.2:40513)
I20260925 03:03:25.648089 23949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "845902c057d54c5ab9ddc861f622125c" candidate_uuid: "5dae02dd94274f54ab9c52ab57a880b0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e8496e8e22c447f9b270e862bbb272a3"
I20260925 03:03:25.648278 24024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "845902c057d54c5ab9ddc861f622125c" candidate_uuid: "5dae02dd94274f54ab9c52ab57a880b0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "1c37142ee9c043d7a5aca01030e1bbf5"
I20260925 03:03:25.648727 23949 raft_consensus.cc:3032] T 845902c057d54c5ab9ddc861f622125c P e8496e8e22c447f9b270e862bbb272a3 [term 2 LEADER]: Stepping down as leader of term 2
I20260925 03:03:25.648813 24024 raft_consensus.cc:3037] T 845902c057d54c5ab9ddc861f622125c P 1c37142ee9c043d7a5aca01030e1bbf5 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:03:25.649101 23949 raft_consensus.cc:729] T 845902c057d54c5ab9ddc861f622125c P e8496e8e22c447f9b270e862bbb272a3 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e8496e8e22c447f9b270e862bbb272a3, State: Running, Role: LEADER
I20260925 03:03:25.649868 23949 consensus_queue.cc:260] T 845902c057d54c5ab9ddc861f622125c P e8496e8e22c447f9b270e862bbb272a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:25.651077 23949 raft_consensus.cc:3037] T 845902c057d54c5ab9ddc861f622125c P e8496e8e22c447f9b270e862bbb272a3 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:03:25.655649 24024 raft_consensus.cc:2454] T 845902c057d54c5ab9ddc861f622125c P 1c37142ee9c043d7a5aca01030e1bbf5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dae02dd94274f54ab9c52ab57a880b0 in term 3.
I20260925 03:03:25.656777 24062 leader_election.cc:302] T 845902c057d54c5ab9ddc861f622125c P 5dae02dd94274f54ab9c52ab57a880b0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c37142ee9c043d7a5aca01030e1bbf5, 5dae02dd94274f54ab9c52ab57a880b0; no voters:
I20260925 03:03:25.657529 23949 raft_consensus.cc:2454] T 845902c057d54c5ab9ddc861f622125c P e8496e8e22c447f9b270e862bbb272a3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dae02dd94274f54ab9c52ab57a880b0 in term 3.
I20260925 03:03:25.657560 24172 raft_consensus.cc:2781] T 845902c057d54c5ab9ddc861f622125c P 5dae02dd94274f54ab9c52ab57a880b0 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 03:03:25.658082 24172 raft_consensus.cc:686] T 845902c057d54c5ab9ddc861f622125c P 5dae02dd94274f54ab9c52ab57a880b0 [term 3 LEADER]: Becoming Leader. State: Replica: 5dae02dd94274f54ab9c52ab57a880b0, State: Running, Role: LEADER
I20260925 03:03:25.658792 24172 consensus_queue.cc:237] T 845902c057d54c5ab9ddc861f622125c P 5dae02dd94274f54ab9c52ab57a880b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:25.666222 23847 catalog_manager.cc:5495] T 845902c057d54c5ab9ddc861f622125c P 5dae02dd94274f54ab9c52ab57a880b0 reported cstate change: term changed from 2 to 3, leader changed from e8496e8e22c447f9b270e862bbb272a3 (127.22.193.1) to 5dae02dd94274f54ab9c52ab57a880b0 (127.22.193.3). New cstate: current_term: 3 leader_uuid: "5dae02dd94274f54ab9c52ab57a880b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } health_report { overall_health: HEALTHY } } }
I20260925 03:03:26.106523 24024 raft_consensus.cc:1261] T f58a83ec6f4149108eb0567cd7817ab5 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 3 FOLLOWER]: Refusing update from remote peer 5dae02dd94274f54ab9c52ab57a880b0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260925 03:03:26.107861 24172 consensus_queue.cc:1035] T f58a83ec6f4149108eb0567cd7817ab5 P 5dae02dd94274f54ab9c52ab57a880b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260925 03:03:26.131040 23948 raft_consensus.cc:1261] T 845902c057d54c5ab9ddc861f622125c P e8496e8e22c447f9b270e862bbb272a3 [term 3 FOLLOWER]: Refusing update from remote peer 5dae02dd94274f54ab9c52ab57a880b0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260925 03:03:26.133267 24023 raft_consensus.cc:1261] T 845902c057d54c5ab9ddc861f622125c P 1c37142ee9c043d7a5aca01030e1bbf5 [term 3 FOLLOWER]: Refusing update from remote peer 5dae02dd94274f54ab9c52ab57a880b0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260925 03:03:26.133267 23948 raft_consensus.cc:1261] T f58a83ec6f4149108eb0567cd7817ab5 P e8496e8e22c447f9b270e862bbb272a3 [term 3 FOLLOWER]: Refusing update from remote peer 5dae02dd94274f54ab9c52ab57a880b0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260925 03:03:26.134958 24249 consensus_queue.cc:1035] T 845902c057d54c5ab9ddc861f622125c P 5dae02dd94274f54ab9c52ab57a880b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260925 03:03:26.135073 24247 consensus_queue.cc:1035] T f58a83ec6f4149108eb0567cd7817ab5 P 5dae02dd94274f54ab9c52ab57a880b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260925 03:03:26.135609 24172 consensus_queue.cc:1035] T 845902c057d54c5ab9ddc861f622125c P 5dae02dd94274f54ab9c52ab57a880b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260925 03:03:26.353700 23300 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260925 03:03:26.359411 23300 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260925 03:03:26.359814 23300 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260925 03:03:27.360749 23300 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260925 03:03:27.366564 23300 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260925 03:03:27.367012 23300 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260925 03:03:28.431104 23300 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260925 03:03:28.439525 23300 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260925 03:03:28.440299 23300 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
W20260925 03:03:28.722518 23878 debug-util.cc:398] Leaking SignalData structure 0x7b080004b840 after lost signal to thread 23303
W20260925 03:03:28.723488 23878 debug-util.cc:398] Leaking SignalData structure 0x7b080004bf20 after lost signal to thread 23881
W20260925 03:03:28.724406 23878 debug-util.cc:398] Leaking SignalData structure 0x7b08000f9d60 after lost signal to thread 23973
W20260925 03:03:28.725396 23878 debug-util.cc:398] Leaking SignalData structure 0x7b080009f4a0 after lost signal to thread 24048
W20260925 03:03:28.726269 23878 debug-util.cc:398] Leaking SignalData structure 0x7b08000ee5e0 after lost signal to thread 24122
I20260925 03:03:29.441423 23300 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260925 03:03:29.447351 23300 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260925 03:03:29.447806 23300 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260925 03:03:30.448675 23300 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260925 03:03:30.454010 23300 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260925 03:03:30.454447 23300 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260925 03:03:31.476171 23300 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260925 03:03:31.484268 23300 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260925 03:03:31.493115 23300 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
W20260925 03:03:31.854465 23878 debug-util.cc:398] Leaking SignalData structure 0x7b08000bf2c0 after lost signal to thread 23303
W20260925 03:03:31.855263 23878 debug-util.cc:398] Leaking SignalData structure 0x7b08000beae0 after lost signal to thread 23881
W20260925 03:03:31.856069 23878 debug-util.cc:398] Leaking SignalData structure 0x7b080004b9a0 after lost signal to thread 23973
W20260925 03:03:31.856809 23878 debug-util.cc:398] Leaking SignalData structure 0x7b08001b6680 after lost signal to thread 24048
W20260925 03:03:31.857473 23878 debug-util.cc:398] Leaking SignalData structure 0x7b08000d42c0 after lost signal to thread 24122
I20260925 03:03:32.495453 23300 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260925 03:03:32.500634 23300 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260925 03:03:32.501051 23300 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260925 03:03:33.501847 23300 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260925 03:03:33.507091 23300 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260925 03:03:33.507452 23300 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260925 03:03:34.508213 23300 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260925 03:03:34.513885 23300 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260925 03:03:34.514334 23300 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260925 03:03:35.544042 23300 auto_leader_rebalancer-test.cc:358] The leader distribution is
I20260925 03:03:35.548568 24098 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f58a83ec6f4149108eb0567cd7817ab5"
dest_uuid: "5dae02dd94274f54ab9c52ab57a880b0"
mode: GRACEFUL
new_leader_uuid: "e8496e8e22c447f9b270e862bbb272a3"
from {username='slave'} at 127.0.0.1:46502
I20260925 03:03:35.549265 24098 raft_consensus.cc:595] T f58a83ec6f4149108eb0567cd7817ab5 P 5dae02dd94274f54ab9c52ab57a880b0 [term 3 LEADER]: Received request to transfer leadership to TS e8496e8e22c447f9b270e862bbb272a3
I20260925 03:03:35.561358 24098 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "845902c057d54c5ab9ddc861f622125c"
dest_uuid: "5dae02dd94274f54ab9c52ab57a880b0"
mode: GRACEFUL
new_leader_uuid: "e8496e8e22c447f9b270e862bbb272a3"
from {username='slave'} at 127.0.0.1:46502
I20260925 03:03:35.561830 24098 raft_consensus.cc:595] T 845902c057d54c5ab9ddc861f622125c P 5dae02dd94274f54ab9c52ab57a880b0 [term 3 LEADER]: Received request to transfer leadership to TS e8496e8e22c447f9b270e862bbb272a3
I20260925 03:03:35.566118 24023 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "400a066077234276876ac579e8353ad2"
dest_uuid: "1c37142ee9c043d7a5aca01030e1bbf5"
mode: GRACEFUL
new_leader_uuid: "e8496e8e22c447f9b270e862bbb272a3"
from {username='slave'} at 127.0.0.1:40134
I20260925 03:03:35.566568 24023 raft_consensus.cc:595] T 400a066077234276876ac579e8353ad2 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 1 LEADER]: Received request to transfer leadership to TS e8496e8e22c447f9b270e862bbb272a3
I20260925 03:03:35.570899 24023 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "78943162e8394d039f00afadd62893f3"
dest_uuid: "1c37142ee9c043d7a5aca01030e1bbf5"
mode: GRACEFUL
new_leader_uuid: "e8496e8e22c447f9b270e862bbb272a3"
from {username='slave'} at 127.0.0.1:40134
I20260925 03:03:35.571341 24023 raft_consensus.cc:595] T 78943162e8394d039f00afadd62893f3 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 2 LEADER]: Received request to transfer leadership to TS e8496e8e22c447f9b270e862bbb272a3
I20260925 03:03:35.575505 24023 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5055d207a0994afbbbc25a2c39455f87"
dest_uuid: "1c37142ee9c043d7a5aca01030e1bbf5"
mode: GRACEFUL
new_leader_uuid: "e8496e8e22c447f9b270e862bbb272a3"
from {username='slave'} at 127.0.0.1:40134
I20260925 03:03:35.576052 24023 raft_consensus.cc:595] T 5055d207a0994afbbbc25a2c39455f87 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 2 LEADER]: Received request to transfer leadership to TS e8496e8e22c447f9b270e862bbb272a3
I20260925 03:03:35.677780 24269 raft_consensus.cc:979] T f58a83ec6f4149108eb0567cd7817ab5 P 5dae02dd94274f54ab9c52ab57a880b0: : Instructing follower e8496e8e22c447f9b270e862bbb272a3 to start an election
I20260925 03:03:35.678184 24269 raft_consensus.cc:1067] T f58a83ec6f4149108eb0567cd7817ab5 P 5dae02dd94274f54ab9c52ab57a880b0 [term 3 LEADER]: Signalling peer e8496e8e22c447f9b270e862bbb272a3 to start an election
I20260925 03:03:35.679754 23949 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f58a83ec6f4149108eb0567cd7817ab5"
dest_uuid: "e8496e8e22c447f9b270e862bbb272a3"
from {username='slave'} at 127.0.0.1:50460
I20260925 03:03:35.680374 23949 raft_consensus.cc:482] T f58a83ec6f4149108eb0567cd7817ab5 P e8496e8e22c447f9b270e862bbb272a3 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 03:03:35.680760 23949 raft_consensus.cc:3037] T f58a83ec6f4149108eb0567cd7817ab5 P e8496e8e22c447f9b270e862bbb272a3 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:03:35.686444 23949 raft_consensus.cc:504] T f58a83ec6f4149108eb0567cd7817ab5 P e8496e8e22c447f9b270e862bbb272a3 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:35.688210 23949 leader_election.cc:288] T f58a83ec6f4149108eb0567cd7817ab5 P e8496e8e22c447f9b270e862bbb272a3 [CANDIDATE]: Term 4 election: Requested vote from peers 1c37142ee9c043d7a5aca01030e1bbf5 (127.22.193.2:40513), 5dae02dd94274f54ab9c52ab57a880b0 (127.22.193.3:46383)
I20260925 03:03:35.689146 24023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f58a83ec6f4149108eb0567cd7817ab5" candidate_uuid: "e8496e8e22c447f9b270e862bbb272a3" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "1c37142ee9c043d7a5aca01030e1bbf5"
I20260925 03:03:35.689227 24098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f58a83ec6f4149108eb0567cd7817ab5" candidate_uuid: "e8496e8e22c447f9b270e862bbb272a3" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "5dae02dd94274f54ab9c52ab57a880b0"
I20260925 03:03:35.689999 24023 raft_consensus.cc:3037] T f58a83ec6f4149108eb0567cd7817ab5 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:03:35.690056 24098 raft_consensus.cc:3032] T f58a83ec6f4149108eb0567cd7817ab5 P 5dae02dd94274f54ab9c52ab57a880b0 [term 3 LEADER]: Stepping down as leader of term 3
I20260925 03:03:35.690457 24098 raft_consensus.cc:729] T f58a83ec6f4149108eb0567cd7817ab5 P 5dae02dd94274f54ab9c52ab57a880b0 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 5dae02dd94274f54ab9c52ab57a880b0, State: Running, Role: LEADER
I20260925 03:03:35.691231 24098 consensus_queue.cc:260] T f58a83ec6f4149108eb0567cd7817ab5 P 5dae02dd94274f54ab9c52ab57a880b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:35.692505 24098 raft_consensus.cc:3037] T f58a83ec6f4149108eb0567cd7817ab5 P 5dae02dd94274f54ab9c52ab57a880b0 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:03:35.693270 24279 raft_consensus.cc:979] T 5055d207a0994afbbbc25a2c39455f87 P 1c37142ee9c043d7a5aca01030e1bbf5: : Instructing follower e8496e8e22c447f9b270e862bbb272a3 to start an election
I20260925 03:03:35.693849 24277 raft_consensus.cc:1067] T 5055d207a0994afbbbc25a2c39455f87 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 2 LEADER]: Signalling peer e8496e8e22c447f9b270e862bbb272a3 to start an election
I20260925 03:03:35.695252 23949 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5055d207a0994afbbbc25a2c39455f87"
dest_uuid: "e8496e8e22c447f9b270e862bbb272a3"
from {username='slave'} at 127.0.0.1:50454
I20260925 03:03:35.695716 23949 raft_consensus.cc:482] T 5055d207a0994afbbbc25a2c39455f87 P e8496e8e22c447f9b270e862bbb272a3 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 03:03:35.696035 23949 raft_consensus.cc:3037] T 5055d207a0994afbbbc25a2c39455f87 P e8496e8e22c447f9b270e862bbb272a3 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:03:35.697399 24023 raft_consensus.cc:2454] T f58a83ec6f4149108eb0567cd7817ab5 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8496e8e22c447f9b270e862bbb272a3 in term 4.
I20260925 03:03:35.698642 23913 leader_election.cc:302] T f58a83ec6f4149108eb0567cd7817ab5 P e8496e8e22c447f9b270e862bbb272a3 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c37142ee9c043d7a5aca01030e1bbf5, e8496e8e22c447f9b270e862bbb272a3; no voters:
I20260925 03:03:35.699422 24098 raft_consensus.cc:2454] T f58a83ec6f4149108eb0567cd7817ab5 P 5dae02dd94274f54ab9c52ab57a880b0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8496e8e22c447f9b270e862bbb272a3 in term 4.
I20260925 03:03:35.699579 24273 raft_consensus.cc:2781] T f58a83ec6f4149108eb0567cd7817ab5 P e8496e8e22c447f9b270e862bbb272a3 [term 4 FOLLOWER]: Leader election won for term 4
I20260925 03:03:35.700776 24273 raft_consensus.cc:686] T f58a83ec6f4149108eb0567cd7817ab5 P e8496e8e22c447f9b270e862bbb272a3 [term 4 LEADER]: Becoming Leader. State: Replica: e8496e8e22c447f9b270e862bbb272a3, State: Running, Role: LEADER
I20260925 03:03:35.701812 24273 consensus_queue.cc:237] T f58a83ec6f4149108eb0567cd7817ab5 P e8496e8e22c447f9b270e862bbb272a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:35.702603 23949 raft_consensus.cc:504] T 5055d207a0994afbbbc25a2c39455f87 P e8496e8e22c447f9b270e862bbb272a3 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:35.704512 23949 leader_election.cc:288] T 5055d207a0994afbbbc25a2c39455f87 P e8496e8e22c447f9b270e862bbb272a3 [CANDIDATE]: Term 3 election: Requested vote from peers 1c37142ee9c043d7a5aca01030e1bbf5 (127.22.193.2:40513), 5dae02dd94274f54ab9c52ab57a880b0 (127.22.193.3:46383)
I20260925 03:03:35.705205 24023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5055d207a0994afbbbc25a2c39455f87" candidate_uuid: "e8496e8e22c447f9b270e862bbb272a3" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "1c37142ee9c043d7a5aca01030e1bbf5"
I20260925 03:03:35.705466 24098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5055d207a0994afbbbc25a2c39455f87" candidate_uuid: "e8496e8e22c447f9b270e862bbb272a3" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "5dae02dd94274f54ab9c52ab57a880b0"
I20260925 03:03:35.705762 24023 raft_consensus.cc:3032] T 5055d207a0994afbbbc25a2c39455f87 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 2 LEADER]: Stepping down as leader of term 2
I20260925 03:03:35.705910 24098 raft_consensus.cc:3037] T 5055d207a0994afbbbc25a2c39455f87 P 5dae02dd94274f54ab9c52ab57a880b0 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:03:35.706053 24023 raft_consensus.cc:729] T 5055d207a0994afbbbc25a2c39455f87 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1c37142ee9c043d7a5aca01030e1bbf5, State: Running, Role: LEADER
I20260925 03:03:35.706878 24023 consensus_queue.cc:260] T 5055d207a0994afbbbc25a2c39455f87 P 1c37142ee9c043d7a5aca01030e1bbf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:35.708036 24023 raft_consensus.cc:3037] T 5055d207a0994afbbbc25a2c39455f87 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:03:35.712339 24098 raft_consensus.cc:2454] T 5055d207a0994afbbbc25a2c39455f87 P 5dae02dd94274f54ab9c52ab57a880b0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8496e8e22c447f9b270e862bbb272a3 in term 3.
I20260925 03:03:35.713903 23915 leader_election.cc:302] T 5055d207a0994afbbbc25a2c39455f87 P e8496e8e22c447f9b270e862bbb272a3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dae02dd94274f54ab9c52ab57a880b0, e8496e8e22c447f9b270e862bbb272a3; no voters:
I20260925 03:03:35.714227 24023 raft_consensus.cc:2454] T 5055d207a0994afbbbc25a2c39455f87 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8496e8e22c447f9b270e862bbb272a3 in term 3.
I20260925 03:03:35.715768 24273 raft_consensus.cc:2781] T 5055d207a0994afbbbc25a2c39455f87 P e8496e8e22c447f9b270e862bbb272a3 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 03:03:35.716158 24273 raft_consensus.cc:686] T 5055d207a0994afbbbc25a2c39455f87 P e8496e8e22c447f9b270e862bbb272a3 [term 3 LEADER]: Becoming Leader. State: Replica: e8496e8e22c447f9b270e862bbb272a3, State: Running, Role: LEADER
I20260925 03:03:35.716894 24273 consensus_queue.cc:237] T 5055d207a0994afbbbc25a2c39455f87 P e8496e8e22c447f9b270e862bbb272a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:35.722151 23845 catalog_manager.cc:5495] T f58a83ec6f4149108eb0567cd7817ab5 P e8496e8e22c447f9b270e862bbb272a3 reported cstate change: term changed from 3 to 4, leader changed from 5dae02dd94274f54ab9c52ab57a880b0 (127.22.193.3) to e8496e8e22c447f9b270e862bbb272a3 (127.22.193.1). New cstate: current_term: 4 leader_uuid: "e8496e8e22c447f9b270e862bbb272a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } health_report { overall_health: UNKNOWN } } }
I20260925 03:03:35.738353 23845 catalog_manager.cc:5495] T 5055d207a0994afbbbc25a2c39455f87 P e8496e8e22c447f9b270e862bbb272a3 reported cstate change: term changed from 2 to 3, leader changed from 1c37142ee9c043d7a5aca01030e1bbf5 (127.22.193.2) to e8496e8e22c447f9b270e862bbb272a3 (127.22.193.1). New cstate: current_term: 3 leader_uuid: "e8496e8e22c447f9b270e862bbb272a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } health_report { overall_health: UNKNOWN } } }
I20260925 03:03:35.782927 24279 raft_consensus.cc:979] T 400a066077234276876ac579e8353ad2 P 1c37142ee9c043d7a5aca01030e1bbf5: : Instructing follower e8496e8e22c447f9b270e862bbb272a3 to start an election
I20260925 03:03:35.783317 24277 raft_consensus.cc:1067] T 400a066077234276876ac579e8353ad2 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 1 LEADER]: Signalling peer e8496e8e22c447f9b270e862bbb272a3 to start an election
I20260925 03:03:35.785056 23949 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "400a066077234276876ac579e8353ad2"
dest_uuid: "e8496e8e22c447f9b270e862bbb272a3"
from {username='slave'} at 127.0.0.1:50454
I20260925 03:03:35.785573 23949 raft_consensus.cc:482] T 400a066077234276876ac579e8353ad2 P e8496e8e22c447f9b270e862bbb272a3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 03:03:35.785897 23949 raft_consensus.cc:3037] T 400a066077234276876ac579e8353ad2 P e8496e8e22c447f9b270e862bbb272a3 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:03:35.791877 23949 raft_consensus.cc:504] T 400a066077234276876ac579e8353ad2 P e8496e8e22c447f9b270e862bbb272a3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:35.793344 23949 leader_election.cc:288] T 400a066077234276876ac579e8353ad2 P e8496e8e22c447f9b270e862bbb272a3 [CANDIDATE]: Term 2 election: Requested vote from peers 1c37142ee9c043d7a5aca01030e1bbf5 (127.22.193.2:40513), 5dae02dd94274f54ab9c52ab57a880b0 (127.22.193.3:46383)
I20260925 03:03:35.794250 24023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "400a066077234276876ac579e8353ad2" candidate_uuid: "e8496e8e22c447f9b270e862bbb272a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1c37142ee9c043d7a5aca01030e1bbf5"
I20260925 03:03:35.794461 24098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "400a066077234276876ac579e8353ad2" candidate_uuid: "e8496e8e22c447f9b270e862bbb272a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5dae02dd94274f54ab9c52ab57a880b0"
I20260925 03:03:35.794852 24023 raft_consensus.cc:3032] T 400a066077234276876ac579e8353ad2 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 1 LEADER]: Stepping down as leader of term 1
I20260925 03:03:35.794903 24098 raft_consensus.cc:3037] T 400a066077234276876ac579e8353ad2 P 5dae02dd94274f54ab9c52ab57a880b0 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:03:35.795243 24023 raft_consensus.cc:729] T 400a066077234276876ac579e8353ad2 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1c37142ee9c043d7a5aca01030e1bbf5, State: Running, Role: LEADER
I20260925 03:03:35.796065 24023 consensus_queue.cc:260] T 400a066077234276876ac579e8353ad2 P 1c37142ee9c043d7a5aca01030e1bbf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:35.797335 24023 raft_consensus.cc:3037] T 400a066077234276876ac579e8353ad2 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:03:35.801565 24098 raft_consensus.cc:2454] T 400a066077234276876ac579e8353ad2 P 5dae02dd94274f54ab9c52ab57a880b0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8496e8e22c447f9b270e862bbb272a3 in term 2.
I20260925 03:03:35.802568 23915 leader_election.cc:302] T 400a066077234276876ac579e8353ad2 P e8496e8e22c447f9b270e862bbb272a3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dae02dd94274f54ab9c52ab57a880b0, e8496e8e22c447f9b270e862bbb272a3; no voters:
I20260925 03:03:35.803316 24273 raft_consensus.cc:2781] T 400a066077234276876ac579e8353ad2 P e8496e8e22c447f9b270e862bbb272a3 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 03:03:35.803820 24273 raft_consensus.cc:686] T 400a066077234276876ac579e8353ad2 P e8496e8e22c447f9b270e862bbb272a3 [term 2 LEADER]: Becoming Leader. State: Replica: e8496e8e22c447f9b270e862bbb272a3, State: Running, Role: LEADER
I20260925 03:03:35.804390 24023 raft_consensus.cc:2454] T 400a066077234276876ac579e8353ad2 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8496e8e22c447f9b270e862bbb272a3 in term 2.
I20260925 03:03:35.804600 24273 consensus_queue.cc:237] T 400a066077234276876ac579e8353ad2 P e8496e8e22c447f9b270e862bbb272a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:35.812285 23845 catalog_manager.cc:5495] T 400a066077234276876ac579e8353ad2 P e8496e8e22c447f9b270e862bbb272a3 reported cstate change: term changed from 1 to 2, leader changed from 1c37142ee9c043d7a5aca01030e1bbf5 (127.22.193.2) to e8496e8e22c447f9b270e862bbb272a3 (127.22.193.1). New cstate: current_term: 2 leader_uuid: "e8496e8e22c447f9b270e862bbb272a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } health_report { overall_health: UNKNOWN } } }
I20260925 03:03:35.856669 24278 raft_consensus.cc:979] T 845902c057d54c5ab9ddc861f622125c P 5dae02dd94274f54ab9c52ab57a880b0: : Instructing follower e8496e8e22c447f9b270e862bbb272a3 to start an election
I20260925 03:03:35.857336 24269 raft_consensus.cc:1067] T 845902c057d54c5ab9ddc861f622125c P 5dae02dd94274f54ab9c52ab57a880b0 [term 3 LEADER]: Signalling peer e8496e8e22c447f9b270e862bbb272a3 to start an election
I20260925 03:03:35.858848 23949 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "845902c057d54c5ab9ddc861f622125c"
dest_uuid: "e8496e8e22c447f9b270e862bbb272a3"
from {username='slave'} at 127.0.0.1:50460
I20260925 03:03:35.859247 23949 raft_consensus.cc:482] T 845902c057d54c5ab9ddc861f622125c P e8496e8e22c447f9b270e862bbb272a3 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 03:03:35.859477 23949 raft_consensus.cc:3037] T 845902c057d54c5ab9ddc861f622125c P e8496e8e22c447f9b270e862bbb272a3 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:03:35.864131 23949 raft_consensus.cc:504] T 845902c057d54c5ab9ddc861f622125c P e8496e8e22c447f9b270e862bbb272a3 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:35.865447 23949 leader_election.cc:288] T 845902c057d54c5ab9ddc861f622125c P e8496e8e22c447f9b270e862bbb272a3 [CANDIDATE]: Term 4 election: Requested vote from peers 1c37142ee9c043d7a5aca01030e1bbf5 (127.22.193.2:40513), 5dae02dd94274f54ab9c52ab57a880b0 (127.22.193.3:46383)
I20260925 03:03:35.866289 24098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "845902c057d54c5ab9ddc861f622125c" candidate_uuid: "e8496e8e22c447f9b270e862bbb272a3" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "5dae02dd94274f54ab9c52ab57a880b0"
I20260925 03:03:35.866289 24023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "845902c057d54c5ab9ddc861f622125c" candidate_uuid: "e8496e8e22c447f9b270e862bbb272a3" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "1c37142ee9c043d7a5aca01030e1bbf5"
I20260925 03:03:35.866948 24098 raft_consensus.cc:3032] T 845902c057d54c5ab9ddc861f622125c P 5dae02dd94274f54ab9c52ab57a880b0 [term 3 LEADER]: Stepping down as leader of term 3
I20260925 03:03:35.866959 24023 raft_consensus.cc:3037] T 845902c057d54c5ab9ddc861f622125c P 1c37142ee9c043d7a5aca01030e1bbf5 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:03:35.867331 24098 raft_consensus.cc:729] T 845902c057d54c5ab9ddc861f622125c P 5dae02dd94274f54ab9c52ab57a880b0 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 5dae02dd94274f54ab9c52ab57a880b0, State: Running, Role: LEADER
I20260925 03:03:35.868193 24098 consensus_queue.cc:260] T 845902c057d54c5ab9ddc861f622125c P 5dae02dd94274f54ab9c52ab57a880b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:35.869598 24098 raft_consensus.cc:3037] T 845902c057d54c5ab9ddc861f622125c P 5dae02dd94274f54ab9c52ab57a880b0 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:03:35.874192 24023 raft_consensus.cc:2454] T 845902c057d54c5ab9ddc861f622125c P 1c37142ee9c043d7a5aca01030e1bbf5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8496e8e22c447f9b270e862bbb272a3 in term 4.
I20260925 03:03:35.875236 23913 leader_election.cc:302] T 845902c057d54c5ab9ddc861f622125c P e8496e8e22c447f9b270e862bbb272a3 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c37142ee9c043d7a5aca01030e1bbf5, e8496e8e22c447f9b270e862bbb272a3; no voters:
I20260925 03:03:35.875885 24098 raft_consensus.cc:2454] T 845902c057d54c5ab9ddc861f622125c P 5dae02dd94274f54ab9c52ab57a880b0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8496e8e22c447f9b270e862bbb272a3 in term 4.
I20260925 03:03:35.875949 24273 raft_consensus.cc:2781] T 845902c057d54c5ab9ddc861f622125c P e8496e8e22c447f9b270e862bbb272a3 [term 4 FOLLOWER]: Leader election won for term 4
I20260925 03:03:35.876425 24273 raft_consensus.cc:686] T 845902c057d54c5ab9ddc861f622125c P e8496e8e22c447f9b270e862bbb272a3 [term 4 LEADER]: Becoming Leader. State: Replica: e8496e8e22c447f9b270e862bbb272a3, State: Running, Role: LEADER
I20260925 03:03:35.877527 24273 consensus_queue.cc:237] T 845902c057d54c5ab9ddc861f622125c P e8496e8e22c447f9b270e862bbb272a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:35.885622 23845 catalog_manager.cc:5495] T 845902c057d54c5ab9ddc861f622125c P e8496e8e22c447f9b270e862bbb272a3 reported cstate change: term changed from 3 to 4, leader changed from 5dae02dd94274f54ab9c52ab57a880b0 (127.22.193.3) to e8496e8e22c447f9b270e862bbb272a3 (127.22.193.1). New cstate: current_term: 4 leader_uuid: "e8496e8e22c447f9b270e862bbb272a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } health_report { overall_health: UNKNOWN } } }
I20260925 03:03:35.988947 24279 raft_consensus.cc:979] T 78943162e8394d039f00afadd62893f3 P 1c37142ee9c043d7a5aca01030e1bbf5: : Instructing follower e8496e8e22c447f9b270e862bbb272a3 to start an election
I20260925 03:03:35.989374 24277 raft_consensus.cc:1067] T 78943162e8394d039f00afadd62893f3 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 2 LEADER]: Signalling peer e8496e8e22c447f9b270e862bbb272a3 to start an election
I20260925 03:03:35.990727 23949 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "78943162e8394d039f00afadd62893f3"
dest_uuid: "e8496e8e22c447f9b270e862bbb272a3"
from {username='slave'} at 127.0.0.1:50454
I20260925 03:03:35.991189 23949 raft_consensus.cc:482] T 78943162e8394d039f00afadd62893f3 P e8496e8e22c447f9b270e862bbb272a3 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 03:03:35.991475 23949 raft_consensus.cc:3037] T 78943162e8394d039f00afadd62893f3 P e8496e8e22c447f9b270e862bbb272a3 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:03:35.995770 23949 raft_consensus.cc:504] T 78943162e8394d039f00afadd62893f3 P e8496e8e22c447f9b270e862bbb272a3 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:35.997169 23949 leader_election.cc:288] T 78943162e8394d039f00afadd62893f3 P e8496e8e22c447f9b270e862bbb272a3 [CANDIDATE]: Term 3 election: Requested vote from peers 1c37142ee9c043d7a5aca01030e1bbf5 (127.22.193.2:40513), 5dae02dd94274f54ab9c52ab57a880b0 (127.22.193.3:46383)
I20260925 03:03:35.998142 24098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78943162e8394d039f00afadd62893f3" candidate_uuid: "e8496e8e22c447f9b270e862bbb272a3" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "5dae02dd94274f54ab9c52ab57a880b0"
I20260925 03:03:35.998198 24023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78943162e8394d039f00afadd62893f3" candidate_uuid: "e8496e8e22c447f9b270e862bbb272a3" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "1c37142ee9c043d7a5aca01030e1bbf5"
I20260925 03:03:35.998685 24098 raft_consensus.cc:3037] T 78943162e8394d039f00afadd62893f3 P 5dae02dd94274f54ab9c52ab57a880b0 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:03:35.998731 24023 raft_consensus.cc:3032] T 78943162e8394d039f00afadd62893f3 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 2 LEADER]: Stepping down as leader of term 2
I20260925 03:03:35.999015 24023 raft_consensus.cc:729] T 78943162e8394d039f00afadd62893f3 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1c37142ee9c043d7a5aca01030e1bbf5, State: Running, Role: LEADER
I20260925 03:03:35.999576 24023 consensus_queue.cc:260] T 78943162e8394d039f00afadd62893f3 P 1c37142ee9c043d7a5aca01030e1bbf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:36.000453 24023 raft_consensus.cc:3037] T 78943162e8394d039f00afadd62893f3 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:03:36.003360 24098 raft_consensus.cc:2454] T 78943162e8394d039f00afadd62893f3 P 5dae02dd94274f54ab9c52ab57a880b0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8496e8e22c447f9b270e862bbb272a3 in term 3.
I20260925 03:03:36.004269 23915 leader_election.cc:302] T 78943162e8394d039f00afadd62893f3 P e8496e8e22c447f9b270e862bbb272a3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dae02dd94274f54ab9c52ab57a880b0, e8496e8e22c447f9b270e862bbb272a3; no voters:
I20260925 03:03:36.004551 24023 raft_consensus.cc:2454] T 78943162e8394d039f00afadd62893f3 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8496e8e22c447f9b270e862bbb272a3 in term 3.
I20260925 03:03:36.005082 24273 raft_consensus.cc:2781] T 78943162e8394d039f00afadd62893f3 P e8496e8e22c447f9b270e862bbb272a3 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 03:03:36.005540 24273 raft_consensus.cc:686] T 78943162e8394d039f00afadd62893f3 P e8496e8e22c447f9b270e862bbb272a3 [term 3 LEADER]: Becoming Leader. State: Replica: e8496e8e22c447f9b270e862bbb272a3, State: Running, Role: LEADER
I20260925 03:03:36.006330 24273 consensus_queue.cc:237] T 78943162e8394d039f00afadd62893f3 P e8496e8e22c447f9b270e862bbb272a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:36.013681 23845 catalog_manager.cc:5495] T 78943162e8394d039f00afadd62893f3 P e8496e8e22c447f9b270e862bbb272a3 reported cstate change: term changed from 2 to 3, leader changed from 1c37142ee9c043d7a5aca01030e1bbf5 (127.22.193.2) to e8496e8e22c447f9b270e862bbb272a3 (127.22.193.1). New cstate: current_term: 3 leader_uuid: "e8496e8e22c447f9b270e862bbb272a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } health_report { overall_health: UNKNOWN } } }
I20260925 03:03:36.131129 24098 raft_consensus.cc:1261] T f58a83ec6f4149108eb0567cd7817ab5 P 5dae02dd94274f54ab9c52ab57a880b0 [term 4 FOLLOWER]: Refusing update from remote peer e8496e8e22c447f9b270e862bbb272a3: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260925 03:03:36.132362 24273 consensus_queue.cc:1035] T f58a83ec6f4149108eb0567cd7817ab5 P e8496e8e22c447f9b270e862bbb272a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260925 03:03:36.143640 24023 raft_consensus.cc:1261] T f58a83ec6f4149108eb0567cd7817ab5 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 4 FOLLOWER]: Refusing update from remote peer e8496e8e22c447f9b270e862bbb272a3: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260925 03:03:36.145154 24280 consensus_queue.cc:1035] T f58a83ec6f4149108eb0567cd7817ab5 P e8496e8e22c447f9b270e862bbb272a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260925 03:03:36.170559 24098 raft_consensus.cc:1261] T 5055d207a0994afbbbc25a2c39455f87 P 5dae02dd94274f54ab9c52ab57a880b0 [term 3 FOLLOWER]: Refusing update from remote peer e8496e8e22c447f9b270e862bbb272a3: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260925 03:03:36.171717 24280 consensus_queue.cc:1035] T 5055d207a0994afbbbc25a2c39455f87 P e8496e8e22c447f9b270e862bbb272a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260925 03:03:36.180359 24023 raft_consensus.cc:1261] T 5055d207a0994afbbbc25a2c39455f87 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 3 FOLLOWER]: Refusing update from remote peer e8496e8e22c447f9b270e862bbb272a3: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260925 03:03:36.183670 24296 consensus_queue.cc:1035] T 5055d207a0994afbbbc25a2c39455f87 P e8496e8e22c447f9b270e862bbb272a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20260925 03:03:36.274523 24023 raft_consensus.cc:1261] T 400a066077234276876ac579e8353ad2 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 2 FOLLOWER]: Refusing update from remote peer e8496e8e22c447f9b270e862bbb272a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260925 03:03:36.275719 24296 consensus_queue.cc:1035] T 400a066077234276876ac579e8353ad2 P e8496e8e22c447f9b270e862bbb272a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260925 03:03:36.285061 24098 raft_consensus.cc:1261] T 400a066077234276876ac579e8353ad2 P 5dae02dd94274f54ab9c52ab57a880b0 [term 2 FOLLOWER]: Refusing update from remote peer e8496e8e22c447f9b270e862bbb272a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260925 03:03:36.286904 24280 consensus_queue.cc:1035] T 400a066077234276876ac579e8353ad2 P e8496e8e22c447f9b270e862bbb272a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260925 03:03:36.392786 24098 raft_consensus.cc:1261] T 78943162e8394d039f00afadd62893f3 P 5dae02dd94274f54ab9c52ab57a880b0 [term 3 FOLLOWER]: Refusing update from remote peer e8496e8e22c447f9b270e862bbb272a3: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260925 03:03:36.393806 24280 consensus_queue.cc:1035] T 78943162e8394d039f00afadd62893f3 P e8496e8e22c447f9b270e862bbb272a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260925 03:03:36.396275 24023 raft_consensus.cc:1261] T 845902c057d54c5ab9ddc861f622125c P 1c37142ee9c043d7a5aca01030e1bbf5 [term 4 FOLLOWER]: Refusing update from remote peer e8496e8e22c447f9b270e862bbb272a3: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260925 03:03:36.398661 24296 consensus_queue.cc:1035] T 845902c057d54c5ab9ddc861f622125c P e8496e8e22c447f9b270e862bbb272a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260925 03:03:36.402655 24022 raft_consensus.cc:1261] T 78943162e8394d039f00afadd62893f3 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 3 FOLLOWER]: Refusing update from remote peer e8496e8e22c447f9b270e862bbb272a3: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260925 03:03:36.408088 24098 raft_consensus.cc:1261] T 845902c057d54c5ab9ddc861f622125c P 5dae02dd94274f54ab9c52ab57a880b0 [term 4 FOLLOWER]: Refusing update from remote peer e8496e8e22c447f9b270e862bbb272a3: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260925 03:03:36.410812 24280 consensus_queue.cc:1035] T 845902c057d54c5ab9ddc861f622125c P e8496e8e22c447f9b270e862bbb272a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260925 03:03:36.411695 24296 consensus_queue.cc:1035] T 78943162e8394d039f00afadd62893f3 P e8496e8e22c447f9b270e862bbb272a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260925 03:03:38.614151 23300 auto_leader_rebalancer-test.cc:373] The leader distribution is
I20260925 03:03:38.614921 23300 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260925 03:03:38.622004 23949 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "48c4fddba997473f8a501507277b8860"
dest_uuid: "e8496e8e22c447f9b270e862bbb272a3"
mode: GRACEFUL
new_leader_uuid: "5dae02dd94274f54ab9c52ab57a880b0"
from {username='slave'} at 127.0.0.1:57306
I20260925 03:03:38.622557 23949 raft_consensus.cc:595] T 48c4fddba997473f8a501507277b8860 P e8496e8e22c447f9b270e862bbb272a3 [term 2 LEADER]: Received request to transfer leadership to TS 5dae02dd94274f54ab9c52ab57a880b0
I20260925 03:03:38.625015 23949 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5055d207a0994afbbbc25a2c39455f87"
dest_uuid: "e8496e8e22c447f9b270e862bbb272a3"
mode: GRACEFUL
new_leader_uuid: "5dae02dd94274f54ab9c52ab57a880b0"
from {username='slave'} at 127.0.0.1:57306
I20260925 03:03:38.625458 23949 raft_consensus.cc:595] T 5055d207a0994afbbbc25a2c39455f87 P e8496e8e22c447f9b270e862bbb272a3 [term 3 LEADER]: Received request to transfer leadership to TS 5dae02dd94274f54ab9c52ab57a880b0
I20260925 03:03:38.627521 23949 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "845902c057d54c5ab9ddc861f622125c"
dest_uuid: "e8496e8e22c447f9b270e862bbb272a3"
mode: GRACEFUL
new_leader_uuid: "1c37142ee9c043d7a5aca01030e1bbf5"
from {username='slave'} at 127.0.0.1:57306
I20260925 03:03:38.627908 23949 raft_consensus.cc:595] T 845902c057d54c5ab9ddc861f622125c P e8496e8e22c447f9b270e862bbb272a3 [term 4 LEADER]: Received request to transfer leadership to TS 1c37142ee9c043d7a5aca01030e1bbf5
I20260925 03:03:38.629887 23949 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c9f3dfa77c7845a49f80417befebb978"
dest_uuid: "e8496e8e22c447f9b270e862bbb272a3"
mode: GRACEFUL
new_leader_uuid: "1c37142ee9c043d7a5aca01030e1bbf5"
from {username='slave'} at 127.0.0.1:57306
I20260925 03:03:38.630283 23949 raft_consensus.cc:595] T c9f3dfa77c7845a49f80417befebb978 P e8496e8e22c447f9b270e862bbb272a3 [term 2 LEADER]: Received request to transfer leadership to TS 1c37142ee9c043d7a5aca01030e1bbf5
I20260925 03:03:38.631359 23300 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260925 03:03:38.631855 23300 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260925 03:03:38.747362 24309 raft_consensus.cc:979] T 48c4fddba997473f8a501507277b8860 P e8496e8e22c447f9b270e862bbb272a3: : Instructing follower 5dae02dd94274f54ab9c52ab57a880b0 to start an election
I20260925 03:03:38.748000 24309 raft_consensus.cc:1067] T 48c4fddba997473f8a501507277b8860 P e8496e8e22c447f9b270e862bbb272a3 [term 2 LEADER]: Signalling peer 5dae02dd94274f54ab9c52ab57a880b0 to start an election
I20260925 03:03:38.749626 24098 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "48c4fddba997473f8a501507277b8860"
dest_uuid: "5dae02dd94274f54ab9c52ab57a880b0"
from {username='slave'} at 127.0.0.1:46490
I20260925 03:03:38.750221 24098 raft_consensus.cc:482] T 48c4fddba997473f8a501507277b8860 P 5dae02dd94274f54ab9c52ab57a880b0 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 03:03:38.750564 24098 raft_consensus.cc:3037] T 48c4fddba997473f8a501507277b8860 P 5dae02dd94274f54ab9c52ab57a880b0 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:03:38.756721 24098 raft_consensus.cc:504] T 48c4fddba997473f8a501507277b8860 P 5dae02dd94274f54ab9c52ab57a880b0 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:38.758929 24098 leader_election.cc:288] T 48c4fddba997473f8a501507277b8860 P 5dae02dd94274f54ab9c52ab57a880b0 [CANDIDATE]: Term 3 election: Requested vote from peers e8496e8e22c447f9b270e862bbb272a3 (127.22.193.1:44975), 1c37142ee9c043d7a5aca01030e1bbf5 (127.22.193.2:40513)
I20260925 03:03:38.760455 23949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48c4fddba997473f8a501507277b8860" candidate_uuid: "5dae02dd94274f54ab9c52ab57a880b0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e8496e8e22c447f9b270e862bbb272a3"
I20260925 03:03:38.760438 24022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48c4fddba997473f8a501507277b8860" candidate_uuid: "5dae02dd94274f54ab9c52ab57a880b0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "1c37142ee9c043d7a5aca01030e1bbf5"
I20260925 03:03:38.761433 23949 raft_consensus.cc:3032] T 48c4fddba997473f8a501507277b8860 P e8496e8e22c447f9b270e862bbb272a3 [term 2 LEADER]: Stepping down as leader of term 2
I20260925 03:03:38.761577 24022 raft_consensus.cc:3037] T 48c4fddba997473f8a501507277b8860 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:03:38.761795 23949 raft_consensus.cc:729] T 48c4fddba997473f8a501507277b8860 P e8496e8e22c447f9b270e862bbb272a3 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e8496e8e22c447f9b270e862bbb272a3, State: Running, Role: LEADER
I20260925 03:03:38.762674 23949 consensus_queue.cc:260] T 48c4fddba997473f8a501507277b8860 P e8496e8e22c447f9b270e862bbb272a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:38.764513 23949 raft_consensus.cc:3037] T 48c4fddba997473f8a501507277b8860 P e8496e8e22c447f9b270e862bbb272a3 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:03:38.770256 24022 raft_consensus.cc:2454] T 48c4fddba997473f8a501507277b8860 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dae02dd94274f54ab9c52ab57a880b0 in term 3.
I20260925 03:03:38.771723 24062 leader_election.cc:302] T 48c4fddba997473f8a501507277b8860 P 5dae02dd94274f54ab9c52ab57a880b0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c37142ee9c043d7a5aca01030e1bbf5, 5dae02dd94274f54ab9c52ab57a880b0; no voters:
I20260925 03:03:38.772629 23949 raft_consensus.cc:2454] T 48c4fddba997473f8a501507277b8860 P e8496e8e22c447f9b270e862bbb272a3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dae02dd94274f54ab9c52ab57a880b0 in term 3.
I20260925 03:03:38.772833 24269 raft_consensus.cc:2781] T 48c4fddba997473f8a501507277b8860 P 5dae02dd94274f54ab9c52ab57a880b0 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 03:03:38.774003 24269 raft_consensus.cc:686] T 48c4fddba997473f8a501507277b8860 P 5dae02dd94274f54ab9c52ab57a880b0 [term 3 LEADER]: Becoming Leader. State: Replica: 5dae02dd94274f54ab9c52ab57a880b0, State: Running, Role: LEADER
I20260925 03:03:38.775242 24269 consensus_queue.cc:237] T 48c4fddba997473f8a501507277b8860 P 5dae02dd94274f54ab9c52ab57a880b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:38.783515 23845 catalog_manager.cc:5495] T 48c4fddba997473f8a501507277b8860 P 5dae02dd94274f54ab9c52ab57a880b0 reported cstate change: term changed from 2 to 3, leader changed from e8496e8e22c447f9b270e862bbb272a3 (127.22.193.1) to 5dae02dd94274f54ab9c52ab57a880b0 (127.22.193.3). New cstate: current_term: 3 leader_uuid: "5dae02dd94274f54ab9c52ab57a880b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } health_report { overall_health: HEALTHY } } }
I20260925 03:03:38.885011 24280 raft_consensus.cc:979] T 845902c057d54c5ab9ddc861f622125c P e8496e8e22c447f9b270e862bbb272a3: : Instructing follower 1c37142ee9c043d7a5aca01030e1bbf5 to start an election
I20260925 03:03:38.885486 24309 raft_consensus.cc:1067] T 845902c057d54c5ab9ddc861f622125c P e8496e8e22c447f9b270e862bbb272a3 [term 4 LEADER]: Signalling peer 1c37142ee9c043d7a5aca01030e1bbf5 to start an election
I20260925 03:03:38.887172 24022 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "845902c057d54c5ab9ddc861f622125c"
dest_uuid: "1c37142ee9c043d7a5aca01030e1bbf5"
from {username='slave'} at 127.0.0.1:40132
I20260925 03:03:38.887704 24022 raft_consensus.cc:482] T 845902c057d54c5ab9ddc861f622125c P 1c37142ee9c043d7a5aca01030e1bbf5 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 03:03:38.888039 24022 raft_consensus.cc:3037] T 845902c057d54c5ab9ddc861f622125c P 1c37142ee9c043d7a5aca01030e1bbf5 [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:03:38.889209 24309 raft_consensus.cc:979] T 5055d207a0994afbbbc25a2c39455f87 P e8496e8e22c447f9b270e862bbb272a3: : Instructing follower 5dae02dd94274f54ab9c52ab57a880b0 to start an election
I20260925 03:03:38.889726 24280 raft_consensus.cc:1067] T 5055d207a0994afbbbc25a2c39455f87 P e8496e8e22c447f9b270e862bbb272a3 [term 3 LEADER]: Signalling peer 5dae02dd94274f54ab9c52ab57a880b0 to start an election
I20260925 03:03:38.891314 24098 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5055d207a0994afbbbc25a2c39455f87"
dest_uuid: "5dae02dd94274f54ab9c52ab57a880b0"
from {username='slave'} at 127.0.0.1:46490
I20260925 03:03:38.891753 24098 raft_consensus.cc:482] T 5055d207a0994afbbbc25a2c39455f87 P 5dae02dd94274f54ab9c52ab57a880b0 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 03:03:38.892072 24098 raft_consensus.cc:3037] T 5055d207a0994afbbbc25a2c39455f87 P 5dae02dd94274f54ab9c52ab57a880b0 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:03:38.894110 24022 raft_consensus.cc:504] T 845902c057d54c5ab9ddc861f622125c P 1c37142ee9c043d7a5aca01030e1bbf5 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:38.896252 24022 leader_election.cc:288] T 845902c057d54c5ab9ddc861f622125c P 1c37142ee9c043d7a5aca01030e1bbf5 [CANDIDATE]: Term 5 election: Requested vote from peers e8496e8e22c447f9b270e862bbb272a3 (127.22.193.1:44975), 5dae02dd94274f54ab9c52ab57a880b0 (127.22.193.3:46383)
I20260925 03:03:38.896728 24098 raft_consensus.cc:504] T 5055d207a0994afbbbc25a2c39455f87 P 5dae02dd94274f54ab9c52ab57a880b0 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:38.897063 23949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "845902c057d54c5ab9ddc861f622125c" candidate_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "e8496e8e22c447f9b270e862bbb272a3"
I20260925 03:03:38.897641 23949 raft_consensus.cc:3032] T 845902c057d54c5ab9ddc861f622125c P e8496e8e22c447f9b270e862bbb272a3 [term 4 LEADER]: Stepping down as leader of term 4
I20260925 03:03:38.897939 23949 raft_consensus.cc:729] T 845902c057d54c5ab9ddc861f622125c P e8496e8e22c447f9b270e862bbb272a3 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: e8496e8e22c447f9b270e862bbb272a3, State: Running, Role: LEADER
I20260925 03:03:38.897747 24096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "845902c057d54c5ab9ddc861f622125c" candidate_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "5dae02dd94274f54ab9c52ab57a880b0"
I20260925 03:03:38.898490 24096 raft_consensus.cc:3037] T 845902c057d54c5ab9ddc861f622125c P 5dae02dd94274f54ab9c52ab57a880b0 [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:03:38.898586 23949 consensus_queue.cc:260] T 845902c057d54c5ab9ddc861f622125c P e8496e8e22c447f9b270e862bbb272a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:38.900063 23949 raft_consensus.cc:3037] T 845902c057d54c5ab9ddc861f622125c P e8496e8e22c447f9b270e862bbb272a3 [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:03:38.901360 24098 leader_election.cc:288] T 5055d207a0994afbbbc25a2c39455f87 P 5dae02dd94274f54ab9c52ab57a880b0 [CANDIDATE]: Term 4 election: Requested vote from peers e8496e8e22c447f9b270e862bbb272a3 (127.22.193.1:44975), 1c37142ee9c043d7a5aca01030e1bbf5 (127.22.193.2:40513)
I20260925 03:03:38.901690 23947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5055d207a0994afbbbc25a2c39455f87" candidate_uuid: "5dae02dd94274f54ab9c52ab57a880b0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "e8496e8e22c447f9b270e862bbb272a3"
I20260925 03:03:38.902163 24023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5055d207a0994afbbbc25a2c39455f87" candidate_uuid: "5dae02dd94274f54ab9c52ab57a880b0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "1c37142ee9c043d7a5aca01030e1bbf5"
I20260925 03:03:38.902313 23947 raft_consensus.cc:3032] T 5055d207a0994afbbbc25a2c39455f87 P e8496e8e22c447f9b270e862bbb272a3 [term 3 LEADER]: Stepping down as leader of term 3
I20260925 03:03:38.902684 23947 raft_consensus.cc:729] T 5055d207a0994afbbbc25a2c39455f87 P e8496e8e22c447f9b270e862bbb272a3 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: e8496e8e22c447f9b270e862bbb272a3, State: Running, Role: LEADER
I20260925 03:03:38.902848 24023 raft_consensus.cc:3037] T 5055d207a0994afbbbc25a2c39455f87 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:03:38.903625 23947 consensus_queue.cc:260] T 5055d207a0994afbbbc25a2c39455f87 P e8496e8e22c447f9b270e862bbb272a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:38.905078 23947 raft_consensus.cc:3037] T 5055d207a0994afbbbc25a2c39455f87 P e8496e8e22c447f9b270e862bbb272a3 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:03:38.906055 24280 raft_consensus.cc:979] T c9f3dfa77c7845a49f80417befebb978 P e8496e8e22c447f9b270e862bbb272a3: : Instructing follower 1c37142ee9c043d7a5aca01030e1bbf5 to start an election
I20260925 03:03:38.906288 24096 raft_consensus.cc:2454] T 845902c057d54c5ab9ddc861f622125c P 5dae02dd94274f54ab9c52ab57a880b0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c37142ee9c043d7a5aca01030e1bbf5 in term 5.
I20260925 03:03:38.906555 24280 raft_consensus.cc:1067] T c9f3dfa77c7845a49f80417befebb978 P e8496e8e22c447f9b270e862bbb272a3 [term 2 LEADER]: Signalling peer 1c37142ee9c043d7a5aca01030e1bbf5 to start an election
I20260925 03:03:38.908435 23990 leader_election.cc:302] T 845902c057d54c5ab9ddc861f622125c P 1c37142ee9c043d7a5aca01030e1bbf5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c37142ee9c043d7a5aca01030e1bbf5, 5dae02dd94274f54ab9c52ab57a880b0; no voters:
I20260925 03:03:38.908443 24022 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c9f3dfa77c7845a49f80417befebb978"
dest_uuid: "1c37142ee9c043d7a5aca01030e1bbf5"
from {username='slave'} at 127.0.0.1:40132
I20260925 03:03:38.909224 24022 raft_consensus.cc:482] T c9f3dfa77c7845a49f80417befebb978 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 03:03:38.909507 24277 raft_consensus.cc:2781] T 845902c057d54c5ab9ddc861f622125c P 1c37142ee9c043d7a5aca01030e1bbf5 [term 5 FOLLOWER]: Leader election won for term 5
I20260925 03:03:38.909583 24022 raft_consensus.cc:3037] T c9f3dfa77c7845a49f80417befebb978 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:03:38.911939 24023 raft_consensus.cc:2454] T 5055d207a0994afbbbc25a2c39455f87 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dae02dd94274f54ab9c52ab57a880b0 in term 4.
I20260925 03:03:38.913090 24062 leader_election.cc:302] T 5055d207a0994afbbbc25a2c39455f87 P 5dae02dd94274f54ab9c52ab57a880b0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c37142ee9c043d7a5aca01030e1bbf5, 5dae02dd94274f54ab9c52ab57a880b0; no voters:
I20260925 03:03:38.912632 23947 raft_consensus.cc:2454] T 5055d207a0994afbbbc25a2c39455f87 P e8496e8e22c447f9b270e862bbb272a3 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dae02dd94274f54ab9c52ab57a880b0 in term 4.
I20260925 03:03:38.916638 24022 raft_consensus.cc:504] T c9f3dfa77c7845a49f80417befebb978 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:38.918421 23949 raft_consensus.cc:2454] T 845902c057d54c5ab9ddc861f622125c P e8496e8e22c447f9b270e862bbb272a3 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c37142ee9c043d7a5aca01030e1bbf5 in term 5.
I20260925 03:03:38.919811 24269 raft_consensus.cc:2781] T 5055d207a0994afbbbc25a2c39455f87 P 5dae02dd94274f54ab9c52ab57a880b0 [term 4 FOLLOWER]: Leader election won for term 4
I20260925 03:03:38.920392 24269 raft_consensus.cc:686] T 5055d207a0994afbbbc25a2c39455f87 P 5dae02dd94274f54ab9c52ab57a880b0 [term 4 LEADER]: Becoming Leader. State: Replica: 5dae02dd94274f54ab9c52ab57a880b0, State: Running, Role: LEADER
I20260925 03:03:38.921743 23946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9f3dfa77c7845a49f80417befebb978" candidate_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e8496e8e22c447f9b270e862bbb272a3"
I20260925 03:03:38.922223 24022 leader_election.cc:288] T c9f3dfa77c7845a49f80417befebb978 P 1c37142ee9c043d7a5aca01030e1bbf5 [CANDIDATE]: Term 3 election: Requested vote from peers e8496e8e22c447f9b270e862bbb272a3 (127.22.193.1:44975), 5dae02dd94274f54ab9c52ab57a880b0 (127.22.193.3:46383)
I20260925 03:03:38.922441 23946 raft_consensus.cc:3032] T c9f3dfa77c7845a49f80417befebb978 P e8496e8e22c447f9b270e862bbb272a3 [term 2 LEADER]: Stepping down as leader of term 2
I20260925 03:03:38.922295 24269 consensus_queue.cc:237] T 5055d207a0994afbbbc25a2c39455f87 P 5dae02dd94274f54ab9c52ab57a880b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:38.923115 24096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9f3dfa77c7845a49f80417befebb978" candidate_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "5dae02dd94274f54ab9c52ab57a880b0"
I20260925 03:03:38.923517 23946 raft_consensus.cc:729] T c9f3dfa77c7845a49f80417befebb978 P e8496e8e22c447f9b270e862bbb272a3 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e8496e8e22c447f9b270e862bbb272a3, State: Running, Role: LEADER
I20260925 03:03:38.923704 24096 raft_consensus.cc:3037] T c9f3dfa77c7845a49f80417befebb978 P 5dae02dd94274f54ab9c52ab57a880b0 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:03:38.924453 23946 consensus_queue.cc:260] T c9f3dfa77c7845a49f80417befebb978 P e8496e8e22c447f9b270e862bbb272a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:38.925297 24277 raft_consensus.cc:686] T 845902c057d54c5ab9ddc861f622125c P 1c37142ee9c043d7a5aca01030e1bbf5 [term 5 LEADER]: Becoming Leader. State: Replica: 1c37142ee9c043d7a5aca01030e1bbf5, State: Running, Role: LEADER
I20260925 03:03:38.925877 23946 raft_consensus.cc:3037] T c9f3dfa77c7845a49f80417befebb978 P e8496e8e22c447f9b270e862bbb272a3 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:03:38.926294 24277 consensus_queue.cc:237] T 845902c057d54c5ab9ddc861f622125c P 1c37142ee9c043d7a5aca01030e1bbf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:38.933166 24096 raft_consensus.cc:2454] T c9f3dfa77c7845a49f80417befebb978 P 5dae02dd94274f54ab9c52ab57a880b0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c37142ee9c043d7a5aca01030e1bbf5 in term 3.
I20260925 03:03:38.933458 23946 raft_consensus.cc:2454] T c9f3dfa77c7845a49f80417befebb978 P e8496e8e22c447f9b270e862bbb272a3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c37142ee9c043d7a5aca01030e1bbf5 in term 3.
I20260925 03:03:38.934384 23990 leader_election.cc:302] T c9f3dfa77c7845a49f80417befebb978 P 1c37142ee9c043d7a5aca01030e1bbf5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c37142ee9c043d7a5aca01030e1bbf5, 5dae02dd94274f54ab9c52ab57a880b0; no voters:
I20260925 03:03:38.935274 24316 raft_consensus.cc:2781] T c9f3dfa77c7845a49f80417befebb978 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 03:03:38.935523 23845 catalog_manager.cc:5495] T 5055d207a0994afbbbc25a2c39455f87 P 5dae02dd94274f54ab9c52ab57a880b0 reported cstate change: term changed from 3 to 4, leader changed from e8496e8e22c447f9b270e862bbb272a3 (127.22.193.1) to 5dae02dd94274f54ab9c52ab57a880b0 (127.22.193.3). New cstate: current_term: 4 leader_uuid: "5dae02dd94274f54ab9c52ab57a880b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } health_report { overall_health: HEALTHY } } }
I20260925 03:03:38.936872 24316 raft_consensus.cc:686] T c9f3dfa77c7845a49f80417befebb978 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 3 LEADER]: Becoming Leader. State: Replica: 1c37142ee9c043d7a5aca01030e1bbf5, State: Running, Role: LEADER
I20260925 03:03:38.937727 24316 consensus_queue.cc:237] T c9f3dfa77c7845a49f80417befebb978 P 1c37142ee9c043d7a5aca01030e1bbf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } }
I20260925 03:03:38.942158 23846 catalog_manager.cc:5495] T 845902c057d54c5ab9ddc861f622125c P 1c37142ee9c043d7a5aca01030e1bbf5 reported cstate change: term changed from 4 to 5, leader changed from e8496e8e22c447f9b270e862bbb272a3 (127.22.193.1) to 1c37142ee9c043d7a5aca01030e1bbf5 (127.22.193.2). New cstate: current_term: 5 leader_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } health_report { overall_health: UNKNOWN } } }
I20260925 03:03:38.954288 23846 catalog_manager.cc:5495] T c9f3dfa77c7845a49f80417befebb978 P 1c37142ee9c043d7a5aca01030e1bbf5 reported cstate change: term changed from 2 to 3, leader changed from e8496e8e22c447f9b270e862bbb272a3 (127.22.193.1) to 1c37142ee9c043d7a5aca01030e1bbf5 (127.22.193.2). New cstate: current_term: 3 leader_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 } health_report { overall_health: UNKNOWN } } }
I20260925 03:03:39.272982 23946 raft_consensus.cc:1261] T 48c4fddba997473f8a501507277b8860 P e8496e8e22c447f9b270e862bbb272a3 [term 3 FOLLOWER]: Refusing update from remote peer 5dae02dd94274f54ab9c52ab57a880b0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260925 03:03:39.274092 24269 consensus_queue.cc:1035] T 48c4fddba997473f8a501507277b8860 P 5dae02dd94274f54ab9c52ab57a880b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260925 03:03:39.283824 24022 raft_consensus.cc:1261] T 48c4fddba997473f8a501507277b8860 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 3 FOLLOWER]: Refusing update from remote peer 5dae02dd94274f54ab9c52ab57a880b0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260925 03:03:39.285578 24310 consensus_queue.cc:1035] T 48c4fddba997473f8a501507277b8860 P 5dae02dd94274f54ab9c52ab57a880b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260925 03:03:39.311939 23946 raft_consensus.cc:1261] T 5055d207a0994afbbbc25a2c39455f87 P e8496e8e22c447f9b270e862bbb272a3 [term 4 FOLLOWER]: Refusing update from remote peer 5dae02dd94274f54ab9c52ab57a880b0: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260925 03:03:39.313159 24310 consensus_queue.cc:1035] T 5055d207a0994afbbbc25a2c39455f87 P 5dae02dd94274f54ab9c52ab57a880b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260925 03:03:39.319759 24022 raft_consensus.cc:1261] T 5055d207a0994afbbbc25a2c39455f87 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 4 FOLLOWER]: Refusing update from remote peer 5dae02dd94274f54ab9c52ab57a880b0: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260925 03:03:39.321882 24310 consensus_queue.cc:1035] T 5055d207a0994afbbbc25a2c39455f87 P 5dae02dd94274f54ab9c52ab57a880b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c37142ee9c043d7a5aca01030e1bbf5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 40513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260925 03:03:39.398231 23946 raft_consensus.cc:1261] T c9f3dfa77c7845a49f80417befebb978 P e8496e8e22c447f9b270e862bbb272a3 [term 3 FOLLOWER]: Refusing update from remote peer 1c37142ee9c043d7a5aca01030e1bbf5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260925 03:03:39.399406 24316 consensus_queue.cc:1035] T c9f3dfa77c7845a49f80417befebb978 P 1c37142ee9c043d7a5aca01030e1bbf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260925 03:03:39.407725 24096 raft_consensus.cc:1261] T c9f3dfa77c7845a49f80417befebb978 P 5dae02dd94274f54ab9c52ab57a880b0 [term 3 FOLLOWER]: Refusing update from remote peer 1c37142ee9c043d7a5aca01030e1bbf5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260925 03:03:39.409555 24277 consensus_queue.cc:1035] T c9f3dfa77c7845a49f80417befebb978 P 1c37142ee9c043d7a5aca01030e1bbf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260925 03:03:39.429029 23946 raft_consensus.cc:1261] T 845902c057d54c5ab9ddc861f622125c P e8496e8e22c447f9b270e862bbb272a3 [term 5 FOLLOWER]: Refusing update from remote peer 1c37142ee9c043d7a5aca01030e1bbf5: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260925 03:03:39.430193 24277 consensus_queue.cc:1035] T 845902c057d54c5ab9ddc861f622125c P 1c37142ee9c043d7a5aca01030e1bbf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8496e8e22c447f9b270e862bbb272a3" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 44975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260925 03:03:39.439200 24096 raft_consensus.cc:1261] T 845902c057d54c5ab9ddc861f622125c P 5dae02dd94274f54ab9c52ab57a880b0 [term 5 FOLLOWER]: Refusing update from remote peer 1c37142ee9c043d7a5aca01030e1bbf5: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260925 03:03:39.440582 24277 consensus_queue.cc:1035] T 845902c057d54c5ab9ddc861f622125c P 1c37142ee9c043d7a5aca01030e1bbf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dae02dd94274f54ab9c52ab57a880b0" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260925 03:03:39.632939 23300 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260925 03:03:39.638119 23300 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260925 03:03:39.638556 23300 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260925 03:03:40.639405 23300 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260925 03:03:40.644567 23300 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260925 03:03:40.645027 23300 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260925 03:03:41.645865 23300 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260925 03:03:41.651172 23300 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260925 03:03:41.651602 23300 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260925 03:03:42.652449 23300 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260925 03:03:42.657691 23300 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260925 03:03:42.658134 23300 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260925 03:03:43.658958 23300 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260925 03:03:43.665114 23300 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260925 03:03:43.665575 23300 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260925 03:03:44.730006 23300 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260925 03:03:44.735946 23300 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260925 03:03:44.736344 23300 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
W20260925 03:03:44.962342 23878 debug-util.cc:398] Leaking SignalData structure 0x7b08000d4820 after lost signal to thread 23303
W20260925 03:03:44.963166 23878 debug-util.cc:398] Leaking SignalData structure 0x7b08001bbf00 after lost signal to thread 23881
W20260925 03:03:44.963972 23878 debug-util.cc:398] Leaking SignalData structure 0x7b08000ee6a0 after lost signal to thread 23973
W20260925 03:03:44.964743 23878 debug-util.cc:398] Leaking SignalData structure 0x7b08001fb340 after lost signal to thread 24048
W20260925 03:03:44.965386 23878 debug-util.cc:398] Leaking SignalData structure 0x7b080009de40 after lost signal to thread 24122
I20260925 03:03:45.737126 23300 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260925 03:03:45.742391 23300 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260925 03:03:45.742867 23300 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260925 03:03:46.743770 23300 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260925 03:03:46.749006 23300 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260925 03:03:46.749468 23300 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260925 03:03:47.750228 23300 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260925 03:03:47.755986 23300 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260925 03:03:47.756385 23300 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260925 03:03:48.791770 23300 auto_leader_rebalancer-test.cc:384] The leader distribution is
I20260925 03:03:48.792428 23300 tablet_server.cc:178] TabletServer@127.22.193.1:0 shutting down...
I20260925 03:03:48.824946 23300 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260925 03:03:48.825925 23300 tablet_replica.cc:330] T 5055d207a0994afbbbc25a2c39455f87 P e8496e8e22c447f9b270e862bbb272a3: stopping tablet replica
I20260925 03:03:48.826908 23300 raft_consensus.cc:2229] T 5055d207a0994afbbbc25a2c39455f87 P e8496e8e22c447f9b270e862bbb272a3 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260925 03:03:48.827772 23300 raft_consensus.cc:2258] T 5055d207a0994afbbbc25a2c39455f87 P e8496e8e22c447f9b270e862bbb272a3 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260925 03:03:48.830582 23300 tablet_replica.cc:330] T f58a83ec6f4149108eb0567cd7817ab5 P e8496e8e22c447f9b270e862bbb272a3: stopping tablet replica
I20260925 03:03:48.831557 23300 raft_consensus.cc:2229] T f58a83ec6f4149108eb0567cd7817ab5 P e8496e8e22c447f9b270e862bbb272a3 [term 4 LEADER]: Raft consensus shutting down.
I20260925 03:03:48.832603 23300 raft_consensus.cc:2258] T f58a83ec6f4149108eb0567cd7817ab5 P e8496e8e22c447f9b270e862bbb272a3 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260925 03:03:48.834544 23300 tablet_replica.cc:330] T be306de199ff4dfb976075219c02ea8e P e8496e8e22c447f9b270e862bbb272a3: stopping tablet replica
I20260925 03:03:48.835114 23300 raft_consensus.cc:2229] T be306de199ff4dfb976075219c02ea8e P e8496e8e22c447f9b270e862bbb272a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 03:03:48.835549 23300 raft_consensus.cc:2258] T be306de199ff4dfb976075219c02ea8e P e8496e8e22c447f9b270e862bbb272a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 03:03:48.837520 23300 tablet_replica.cc:330] T 400a066077234276876ac579e8353ad2 P e8496e8e22c447f9b270e862bbb272a3: stopping tablet replica
I20260925 03:03:48.838033 23300 raft_consensus.cc:2229] T 400a066077234276876ac579e8353ad2 P e8496e8e22c447f9b270e862bbb272a3 [term 2 LEADER]: Raft consensus shutting down.
I20260925 03:03:48.839118 23300 raft_consensus.cc:2258] T 400a066077234276876ac579e8353ad2 P e8496e8e22c447f9b270e862bbb272a3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260925 03:03:48.840847 23300 tablet_replica.cc:330] T 845902c057d54c5ab9ddc861f622125c P e8496e8e22c447f9b270e862bbb272a3: stopping tablet replica
I20260925 03:03:48.841301 23300 raft_consensus.cc:2229] T 845902c057d54c5ab9ddc861f622125c P e8496e8e22c447f9b270e862bbb272a3 [term 5 FOLLOWER]: Raft consensus shutting down.
I20260925 03:03:48.841717 23300 raft_consensus.cc:2258] T 845902c057d54c5ab9ddc861f622125c P e8496e8e22c447f9b270e862bbb272a3 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260925 03:03:48.843495 23300 tablet_replica.cc:330] T c9f3dfa77c7845a49f80417befebb978 P e8496e8e22c447f9b270e862bbb272a3: stopping tablet replica
I20260925 03:03:48.843972 23300 raft_consensus.cc:2229] T c9f3dfa77c7845a49f80417befebb978 P e8496e8e22c447f9b270e862bbb272a3 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260925 03:03:48.844417 23300 raft_consensus.cc:2258] T c9f3dfa77c7845a49f80417befebb978 P e8496e8e22c447f9b270e862bbb272a3 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260925 03:03:48.846374 23300 tablet_replica.cc:330] T 5ac7083552944834bb61700cfbe60faa P e8496e8e22c447f9b270e862bbb272a3: stopping tablet replica
I20260925 03:03:48.846803 23300 raft_consensus.cc:2229] T 5ac7083552944834bb61700cfbe60faa P e8496e8e22c447f9b270e862bbb272a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 03:03:48.847221 23300 raft_consensus.cc:2258] T 5ac7083552944834bb61700cfbe60faa P e8496e8e22c447f9b270e862bbb272a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 03:03:48.848927 23300 tablet_replica.cc:330] T 48c4fddba997473f8a501507277b8860 P e8496e8e22c447f9b270e862bbb272a3: stopping tablet replica
I20260925 03:03:48.849433 23300 raft_consensus.cc:2229] T 48c4fddba997473f8a501507277b8860 P e8496e8e22c447f9b270e862bbb272a3 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260925 03:03:48.849936 23300 raft_consensus.cc:2258] T 48c4fddba997473f8a501507277b8860 P e8496e8e22c447f9b270e862bbb272a3 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260925 03:03:48.851750 23300 tablet_replica.cc:330] T 78943162e8394d039f00afadd62893f3 P e8496e8e22c447f9b270e862bbb272a3: stopping tablet replica
I20260925 03:03:48.852209 23300 raft_consensus.cc:2229] T 78943162e8394d039f00afadd62893f3 P e8496e8e22c447f9b270e862bbb272a3 [term 3 LEADER]: Raft consensus shutting down.
I20260925 03:03:48.853176 23300 raft_consensus.cc:2258] T 78943162e8394d039f00afadd62893f3 P e8496e8e22c447f9b270e862bbb272a3 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260925 03:03:48.854863 23300 tablet_replica.cc:330] T 7f1e503d876c45fa853edeaf3b836180 P e8496e8e22c447f9b270e862bbb272a3: stopping tablet replica
I20260925 03:03:48.855258 23300 raft_consensus.cc:2229] T 7f1e503d876c45fa853edeaf3b836180 P e8496e8e22c447f9b270e862bbb272a3 [term 2 LEADER]: Raft consensus shutting down.
I20260925 03:03:48.856225 23300 raft_consensus.cc:2258] T 7f1e503d876c45fa853edeaf3b836180 P e8496e8e22c447f9b270e862bbb272a3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260925 03:03:48.888947 23300 tablet_server.cc:194] TabletServer@127.22.193.1:0 shutdown complete.
I20260925 03:03:48.910002 23300 tablet_server.cc:178] TabletServer@127.22.193.2:0 shutting down...
I20260925 03:03:48.936031 23300 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260925 03:03:48.936781 23300 tablet_replica.cc:330] T 48c4fddba997473f8a501507277b8860 P 1c37142ee9c043d7a5aca01030e1bbf5: stopping tablet replica
I20260925 03:03:48.937469 23300 raft_consensus.cc:2229] T 48c4fddba997473f8a501507277b8860 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260925 03:03:48.937944 23300 raft_consensus.cc:2258] T 48c4fddba997473f8a501507277b8860 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260925 03:03:48.939762 23300 tablet_replica.cc:330] T 78943162e8394d039f00afadd62893f3 P 1c37142ee9c043d7a5aca01030e1bbf5: stopping tablet replica
I20260925 03:03:48.940312 23300 raft_consensus.cc:2229] T 78943162e8394d039f00afadd62893f3 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260925 03:03:48.940800 23300 raft_consensus.cc:2258] T 78943162e8394d039f00afadd62893f3 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260925 03:03:48.942591 23300 tablet_replica.cc:330] T 400a066077234276876ac579e8353ad2 P 1c37142ee9c043d7a5aca01030e1bbf5: stopping tablet replica
I20260925 03:03:48.943137 23300 raft_consensus.cc:2229] T 400a066077234276876ac579e8353ad2 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260925 03:03:48.943574 23300 raft_consensus.cc:2258] T 400a066077234276876ac579e8353ad2 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260925 03:03:48.945410 23300 tablet_replica.cc:330] T 845902c057d54c5ab9ddc861f622125c P 1c37142ee9c043d7a5aca01030e1bbf5: stopping tablet replica
I20260925 03:03:48.945904 23300 raft_consensus.cc:2229] T 845902c057d54c5ab9ddc861f622125c P 1c37142ee9c043d7a5aca01030e1bbf5 [term 5 LEADER]: Raft consensus shutting down.
I20260925 03:03:48.946974 23300 raft_consensus.cc:2258] T 845902c057d54c5ab9ddc861f622125c P 1c37142ee9c043d7a5aca01030e1bbf5 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260925 03:03:48.948937 23300 tablet_replica.cc:330] T 5055d207a0994afbbbc25a2c39455f87 P 1c37142ee9c043d7a5aca01030e1bbf5: stopping tablet replica
I20260925 03:03:48.949396 23300 raft_consensus.cc:2229] T 5055d207a0994afbbbc25a2c39455f87 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260925 03:03:48.949901 23300 raft_consensus.cc:2258] T 5055d207a0994afbbbc25a2c39455f87 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260925 03:03:48.951654 23300 tablet_replica.cc:330] T f58a83ec6f4149108eb0567cd7817ab5 P 1c37142ee9c043d7a5aca01030e1bbf5: stopping tablet replica
I20260925 03:03:48.952123 23300 raft_consensus.cc:2229] T f58a83ec6f4149108eb0567cd7817ab5 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260925 03:03:48.952497 23300 raft_consensus.cc:2258] T f58a83ec6f4149108eb0567cd7817ab5 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260925 03:03:48.954177 23300 tablet_replica.cc:330] T be306de199ff4dfb976075219c02ea8e P 1c37142ee9c043d7a5aca01030e1bbf5: stopping tablet replica
I20260925 03:03:48.954603 23300 raft_consensus.cc:2229] T be306de199ff4dfb976075219c02ea8e P 1c37142ee9c043d7a5aca01030e1bbf5 [term 1 LEADER]: Raft consensus shutting down.
I20260925 03:03:48.955596 23300 raft_consensus.cc:2258] T be306de199ff4dfb976075219c02ea8e P 1c37142ee9c043d7a5aca01030e1bbf5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 03:03:48.957352 23300 tablet_replica.cc:330] T c9f3dfa77c7845a49f80417befebb978 P 1c37142ee9c043d7a5aca01030e1bbf5: stopping tablet replica
I20260925 03:03:48.957787 23300 raft_consensus.cc:2229] T c9f3dfa77c7845a49f80417befebb978 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 3 LEADER]: Raft consensus shutting down.
I20260925 03:03:48.958618 23300 raft_consensus.cc:2258] T c9f3dfa77c7845a49f80417befebb978 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260925 03:03:48.960299 23300 tablet_replica.cc:330] T 5ac7083552944834bb61700cfbe60faa P 1c37142ee9c043d7a5aca01030e1bbf5: stopping tablet replica
I20260925 03:03:48.960793 23300 raft_consensus.cc:2229] T 5ac7083552944834bb61700cfbe60faa P 1c37142ee9c043d7a5aca01030e1bbf5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 03:03:48.961170 23300 raft_consensus.cc:2258] T 5ac7083552944834bb61700cfbe60faa P 1c37142ee9c043d7a5aca01030e1bbf5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 03:03:48.962779 23300 tablet_replica.cc:330] T 7f1e503d876c45fa853edeaf3b836180 P 1c37142ee9c043d7a5aca01030e1bbf5: stopping tablet replica
I20260925 03:03:48.963236 23300 raft_consensus.cc:2229] T 7f1e503d876c45fa853edeaf3b836180 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260925 03:03:48.963672 23300 raft_consensus.cc:2258] T 7f1e503d876c45fa853edeaf3b836180 P 1c37142ee9c043d7a5aca01030e1bbf5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260925 03:03:48.992398 23300 tablet_server.cc:194] TabletServer@127.22.193.2:0 shutdown complete.
W20260925 03:03:48.996731 24062 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.193.1:44975: connect: Connection refused (error 111) [suppressed 5 similar messages]
W20260925 03:03:49.000090 24062 consensus_peers.cc:487] T 48c4fddba997473f8a501507277b8860 P 5dae02dd94274f54ab9c52ab57a880b0 -> Peer e8496e8e22c447f9b270e862bbb272a3 (127.22.193.1:44975): Couldn't send request to peer e8496e8e22c447f9b270e862bbb272a3. Status: Network error: Client connection negotiation failed: client connection to 127.22.193.1:44975: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 03:03:49.011375 23300 tablet_server.cc:178] TabletServer@127.22.193.3:0 shutting down...
I20260925 03:03:49.036445 23300 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260925 03:03:49.037143 23300 tablet_replica.cc:330] T 78943162e8394d039f00afadd62893f3 P 5dae02dd94274f54ab9c52ab57a880b0: stopping tablet replica
I20260925 03:03:49.037765 23300 raft_consensus.cc:2229] T 78943162e8394d039f00afadd62893f3 P 5dae02dd94274f54ab9c52ab57a880b0 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260925 03:03:49.038367 23300 raft_consensus.cc:2258] T 78943162e8394d039f00afadd62893f3 P 5dae02dd94274f54ab9c52ab57a880b0 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260925 03:03:49.040066 23300 tablet_replica.cc:330] T 48c4fddba997473f8a501507277b8860 P 5dae02dd94274f54ab9c52ab57a880b0: stopping tablet replica
I20260925 03:03:49.040560 23300 raft_consensus.cc:2229] T 48c4fddba997473f8a501507277b8860 P 5dae02dd94274f54ab9c52ab57a880b0 [term 3 LEADER]: Raft consensus shutting down.
I20260925 03:03:49.041419 23300 raft_consensus.cc:2258] T 48c4fddba997473f8a501507277b8860 P 5dae02dd94274f54ab9c52ab57a880b0 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260925 03:03:49.043179 23300 tablet_replica.cc:330] T 7f1e503d876c45fa853edeaf3b836180 P 5dae02dd94274f54ab9c52ab57a880b0: stopping tablet replica
I20260925 03:03:49.043668 23300 raft_consensus.cc:2229] T 7f1e503d876c45fa853edeaf3b836180 P 5dae02dd94274f54ab9c52ab57a880b0 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260925 03:03:49.044091 23300 raft_consensus.cc:2258] T 7f1e503d876c45fa853edeaf3b836180 P 5dae02dd94274f54ab9c52ab57a880b0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260925 03:03:49.045884 23300 tablet_replica.cc:330] T be306de199ff4dfb976075219c02ea8e P 5dae02dd94274f54ab9c52ab57a880b0: stopping tablet replica
I20260925 03:03:49.046377 23300 raft_consensus.cc:2229] T be306de199ff4dfb976075219c02ea8e P 5dae02dd94274f54ab9c52ab57a880b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 03:03:49.046779 23300 raft_consensus.cc:2258] T be306de199ff4dfb976075219c02ea8e P 5dae02dd94274f54ab9c52ab57a880b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 03:03:49.048352 23300 tablet_replica.cc:330] T f58a83ec6f4149108eb0567cd7817ab5 P 5dae02dd94274f54ab9c52ab57a880b0: stopping tablet replica
I20260925 03:03:49.048861 23300 raft_consensus.cc:2229] T f58a83ec6f4149108eb0567cd7817ab5 P 5dae02dd94274f54ab9c52ab57a880b0 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260925 03:03:49.049320 23300 raft_consensus.cc:2258] T f58a83ec6f4149108eb0567cd7817ab5 P 5dae02dd94274f54ab9c52ab57a880b0 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260925 03:03:49.051093 23300 tablet_replica.cc:330] T 400a066077234276876ac579e8353ad2 P 5dae02dd94274f54ab9c52ab57a880b0: stopping tablet replica
I20260925 03:03:49.051556 23300 raft_consensus.cc:2229] T 400a066077234276876ac579e8353ad2 P 5dae02dd94274f54ab9c52ab57a880b0 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260925 03:03:49.052026 23300 raft_consensus.cc:2258] T 400a066077234276876ac579e8353ad2 P 5dae02dd94274f54ab9c52ab57a880b0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260925 03:03:49.053727 23300 tablet_replica.cc:330] T 845902c057d54c5ab9ddc861f622125c P 5dae02dd94274f54ab9c52ab57a880b0: stopping tablet replica
I20260925 03:03:49.054200 23300 raft_consensus.cc:2229] T 845902c057d54c5ab9ddc861f622125c P 5dae02dd94274f54ab9c52ab57a880b0 [term 5 FOLLOWER]: Raft consensus shutting down.
I20260925 03:03:49.054616 23300 raft_consensus.cc:2258] T 845902c057d54c5ab9ddc861f622125c P 5dae02dd94274f54ab9c52ab57a880b0 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260925 03:03:49.056241 23300 tablet_replica.cc:330] T 5055d207a0994afbbbc25a2c39455f87 P 5dae02dd94274f54ab9c52ab57a880b0: stopping tablet replica
I20260925 03:03:49.056804 23300 raft_consensus.cc:2229] T 5055d207a0994afbbbc25a2c39455f87 P 5dae02dd94274f54ab9c52ab57a880b0 [term 4 LEADER]: Raft consensus shutting down.
I20260925 03:03:49.057667 23300 raft_consensus.cc:2258] T 5055d207a0994afbbbc25a2c39455f87 P 5dae02dd94274f54ab9c52ab57a880b0 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260925 03:03:49.059304 23300 tablet_replica.cc:330] T c9f3dfa77c7845a49f80417befebb978 P 5dae02dd94274f54ab9c52ab57a880b0: stopping tablet replica
I20260925 03:03:49.059721 23300 raft_consensus.cc:2229] T c9f3dfa77c7845a49f80417befebb978 P 5dae02dd94274f54ab9c52ab57a880b0 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260925 03:03:49.060094 23300 raft_consensus.cc:2258] T c9f3dfa77c7845a49f80417befebb978 P 5dae02dd94274f54ab9c52ab57a880b0 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260925 03:03:49.061661 23300 tablet_replica.cc:330] T 5ac7083552944834bb61700cfbe60faa P 5dae02dd94274f54ab9c52ab57a880b0: stopping tablet replica
I20260925 03:03:49.062067 23300 raft_consensus.cc:2229] T 5ac7083552944834bb61700cfbe60faa P 5dae02dd94274f54ab9c52ab57a880b0 [term 1 LEADER]: Raft consensus shutting down.
I20260925 03:03:49.062923 23300 raft_consensus.cc:2258] T 5ac7083552944834bb61700cfbe60faa P 5dae02dd94274f54ab9c52ab57a880b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 03:03:49.092180 23300 tablet_server.cc:194] TabletServer@127.22.193.3:0 shutdown complete.
I20260925 03:03:49.110836 23300 master.cc:537] Master@127.22.193.62:33513 shutting down...
I20260925 03:03:49.131556 23300 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 06b4bc18a82b4b8fb4a33975c07d3a12 [term 1 LEADER]: Raft consensus shutting down.
I20260925 03:03:49.132192 23300 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 06b4bc18a82b4b8fb4a33975c07d3a12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 03:03:49.132519 23300 tablet_replica.cc:330] T 00000000000000000000000000000000 P 06b4bc18a82b4b8fb4a33975c07d3a12: stopping tablet replica
I20260925 03:03:49.173808 23300 master.cc:552] Master@127.22.193.62:33513 shutdown complete.
1c37142ee9c043d7a5aca01030e1bbf5 4
5dae02dd94274f54ab9c52ab57a880b0 1
e8496e8e22c447f9b270e862bbb272a3 5
1c37142ee9c043d7a5aca01030e1bbf5 4
5dae02dd94274f54ab9c52ab57a880b0 3
e8496e8e22c447f9b270e862bbb272a3 3
1c37142ee9c043d7a5aca01030e1bbf5 1
5dae02dd94274f54ab9c52ab57a880b0 1
e8496e8e22c447f9b270e862bbb272a3 8
1c37142ee9c043d7a5aca01030e1bbf5 3
5dae02dd94274f54ab9c52ab57a880b0 3
e8496e8e22c447f9b270e862bbb272a3 4
[ OK ] LeaderRebalancerTest.FunctionalTestForNotDivided (28800 ms)
[ RUN ] LeaderRebalancerTest.AddTserver
I20260925 03:03:49.213972 23300 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.22.193.62:40353
I20260925 03:03:49.215070 23300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 03:03:49.220908 24347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:03:49.221022 24346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:03:49.221797 24349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 03:03:49.223055 23300 server_base.cc:1034] running on GCE node
I20260925 03:03:49.224210 23300 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260925 03:03:49.224409 23300 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260925 03:03:49.224586 23300 hybrid_clock.cc:648] HybridClock initialized: now 1790305429224570 us; error 0 us; skew 500 ppm
I20260925 03:03:49.225147 23300 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 03:03:49.232771 23300 webserver.cc:460] Webserver started at http://127.22.193.62:34779/ using document root <none> and password file <none>
I20260925 03:03:49.233314 23300 fs_manager.cc:359] Metadata directory not provided
I20260925 03:03:49.233490 23300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 03:03:49.233771 23300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 03:03:49.234989 23300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklAj8hX/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790305368086980-23300-0/minicluster-data/master-0-root/instance:
uuid: "2d92cc0732c4490eb7d98d353826cce1"
format_stamp: "Formatted at 2026-09-25 03:03:49 on dist-test-slave-cswq"
I20260925 03:03:49.240890 23300 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 03:03:49.245630 24354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 03:03:49.246428 23300 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 03:03:49.246692 23300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklAj8hX/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790305368086980-23300-0/minicluster-data/master-0-root
uuid: "2d92cc0732c4490eb7d98d353826cce1"
format_stamp: "Formatted at 2026-09-25 03:03:49 on dist-test-slave-cswq"
I20260925 03:03:49.246945 23300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklAj8hX/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790305368086980-23300-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-tasklAj8hX/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790305368086980-23300-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-tasklAj8hX/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790305368086980-23300-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 03:03:49.267340 23300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 03:03:49.268642 23300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 03:03:49.306355 23300 rpc_server.cc:307] RPC server started. Bound to: 127.22.193.62:40353
I20260925 03:03:49.306459 24405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.193.62:40353 every 8 connection(s)
I20260925 03:03:49.310319 24406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:49.320055 24406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d92cc0732c4490eb7d98d353826cce1: Bootstrap starting.
I20260925 03:03:49.324391 24406 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d92cc0732c4490eb7d98d353826cce1: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:49.328627 24406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d92cc0732c4490eb7d98d353826cce1: No bootstrap required, opened a new log
I20260925 03:03:49.330658 24406 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d92cc0732c4490eb7d98d353826cce1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d92cc0732c4490eb7d98d353826cce1" member_type: VOTER }
I20260925 03:03:49.331142 24406 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2d92cc0732c4490eb7d98d353826cce1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:49.331310 24406 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d92cc0732c4490eb7d98d353826cce1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d92cc0732c4490eb7d98d353826cce1, State: Initialized, Role: FOLLOWER
I20260925 03:03:49.331962 24406 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d92cc0732c4490eb7d98d353826cce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d92cc0732c4490eb7d98d353826cce1" member_type: VOTER }
I20260925 03:03:49.332417 24406 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2d92cc0732c4490eb7d98d353826cce1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 03:03:49.332656 24406 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d92cc0732c4490eb7d98d353826cce1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 03:03:49.332965 24406 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d92cc0732c4490eb7d98d353826cce1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:49.337044 24406 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d92cc0732c4490eb7d98d353826cce1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d92cc0732c4490eb7d98d353826cce1" member_type: VOTER }
I20260925 03:03:49.337543 24406 leader_election.cc:302] T 00000000000000000000000000000000 P 2d92cc0732c4490eb7d98d353826cce1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d92cc0732c4490eb7d98d353826cce1; no voters:
I20260925 03:03:49.338722 24406 leader_election.cc:288] T 00000000000000000000000000000000 P 2d92cc0732c4490eb7d98d353826cce1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 03:03:49.339107 24409 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d92cc0732c4490eb7d98d353826cce1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:03:49.340477 24409 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2d92cc0732c4490eb7d98d353826cce1 [term 1 LEADER]: Becoming Leader. State: Replica: 2d92cc0732c4490eb7d98d353826cce1, State: Running, Role: LEADER
I20260925 03:03:49.341109 24409 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d92cc0732c4490eb7d98d353826cce1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d92cc0732c4490eb7d98d353826cce1" member_type: VOTER }
I20260925 03:03:49.341643 24406 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d92cc0732c4490eb7d98d353826cce1 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 03:03:49.346313 24410 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d92cc0732c4490eb7d98d353826cce1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2d92cc0732c4490eb7d98d353826cce1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d92cc0732c4490eb7d98d353826cce1" member_type: VOTER } }
I20260925 03:03:49.347509 24411 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d92cc0732c4490eb7d98d353826cce1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d92cc0732c4490eb7d98d353826cce1. Latest consensus state: current_term: 1 leader_uuid: "2d92cc0732c4490eb7d98d353826cce1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d92cc0732c4490eb7d98d353826cce1" member_type: VOTER } }
I20260925 03:03:49.347903 24410 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d92cc0732c4490eb7d98d353826cce1 [sys.catalog]: This master's current role is: LEADER
I20260925 03:03:49.348279 24411 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d92cc0732c4490eb7d98d353826cce1 [sys.catalog]: This master's current role is: LEADER
I20260925 03:03:49.350019 24419 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 03:03:49.357394 24419 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 03:03:49.358526 23300 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260925 03:03:49.366235 24419 catalog_manager.cc:1347] Generated new cluster ID: f54e42b7337549a189c3d97042cfeebb
I20260925 03:03:49.366504 24419 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 03:03:49.388339 24419 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 03:03:49.390080 24419 catalog_manager.cc:1504] Loading token signing keys...
I20260925 03:03:49.408447 24419 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2d92cc0732c4490eb7d98d353826cce1: Generated new TSK 0
I20260925 03:03:49.409111 24419 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 03:03:49.425465 23300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 03:03:49.431530 24427 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:03:49.432318 24428 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:03:49.433769 24430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 03:03:49.434306 23300 server_base.cc:1034] running on GCE node
I20260925 03:03:49.435156 23300 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260925 03:03:49.435333 23300 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260925 03:03:49.435451 23300 hybrid_clock.cc:648] HybridClock initialized: now 1790305429435442 us; error 0 us; skew 500 ppm
I20260925 03:03:49.435945 23300 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 03:03:49.438339 23300 webserver.cc:460] Webserver started at http://127.22.193.1:37987/ using document root <none> and password file <none>
I20260925 03:03:49.438761 23300 fs_manager.cc:359] Metadata directory not provided
I20260925 03:03:49.438905 23300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 03:03:49.439113 23300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 03:03:49.440165 23300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklAj8hX/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790305368086980-23300-0/minicluster-data/ts-0-root/instance:
uuid: "869b0a8575b54d08ae2817fad5dd42d5"
format_stamp: "Formatted at 2026-09-25 03:03:49 on dist-test-slave-cswq"
I20260925 03:03:49.444905 23300 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 03:03:49.448182 24435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 03:03:49.448997 23300 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 03:03:49.449294 23300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklAj8hX/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790305368086980-23300-0/minicluster-data/ts-0-root
uuid: "869b0a8575b54d08ae2817fad5dd42d5"
format_stamp: "Formatted at 2026-09-25 03:03:49 on dist-test-slave-cswq"
I20260925 03:03:49.449582 23300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklAj8hX/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790305368086980-23300-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-tasklAj8hX/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790305368086980-23300-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-tasklAj8hX/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790305368086980-23300-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 03:03:49.465269 23300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 03:03:49.466463 23300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 03:03:49.467957 23300 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 03:03:49.470379 23300 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 03:03:49.470578 23300 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 03:03:49.470811 23300 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 03:03:49.470981 23300 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 03:03:49.510263 23300 rpc_server.cc:307] RPC server started. Bound to: 127.22.193.1:42187
I20260925 03:03:49.510367 24497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.193.1:42187 every 8 connection(s)
I20260925 03:03:49.514946 23300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 03:03:49.522447 24502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:03:49.526084 24505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 03:03:49.526332 23300 server_base.cc:1034] running on GCE node
W20260925 03:03:49.526818 24503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 03:03:49.528154 23300 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260925 03:03:49.528363 23300 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260925 03:03:49.528601 23300 hybrid_clock.cc:648] HybridClock initialized: now 1790305429528582 us; error 0 us; skew 500 ppm
I20260925 03:03:49.529328 23300 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 03:03:49.529901 24498 heartbeater.cc:347] Connected to a master server at 127.22.193.62:40353
I20260925 03:03:49.530249 24498 heartbeater.cc:464] Registering TS with master...
I20260925 03:03:49.531081 24498 heartbeater.cc:511] Master 127.22.193.62:40353 requested a full tablet report, sending...
I20260925 03:03:49.532126 23300 webserver.cc:460] Webserver started at http://127.22.193.2:33153/ using document root <none> and password file <none>
I20260925 03:03:49.532604 23300 fs_manager.cc:359] Metadata directory not provided
I20260925 03:03:49.532800 23300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 03:03:49.533047 23300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 03:03:49.533107 24371 ts_manager.cc:194] Registered new tserver with Master: 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187)
I20260925 03:03:49.534216 23300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklAj8hX/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790305368086980-23300-0/minicluster-data/ts-1-root/instance:
uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
format_stamp: "Formatted at 2026-09-25 03:03:49 on dist-test-slave-cswq"
I20260925 03:03:49.535055 24371 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39368
I20260925 03:03:49.538923 23300 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 03:03:49.542501 24510 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 03:03:49.543382 23300 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 03:03:49.543668 23300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklAj8hX/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790305368086980-23300-0/minicluster-data/ts-1-root
uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
format_stamp: "Formatted at 2026-09-25 03:03:49 on dist-test-slave-cswq"
I20260925 03:03:49.543938 23300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklAj8hX/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790305368086980-23300-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-tasklAj8hX/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790305368086980-23300-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-tasklAj8hX/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790305368086980-23300-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 03:03:49.569476 23300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 03:03:49.570657 23300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 03:03:49.572111 23300 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 03:03:49.574258 23300 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 03:03:49.574441 23300 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 03:03:49.574666 23300 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 03:03:49.574795 23300 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 03:03:49.615842 23300 rpc_server.cc:307] RPC server started. Bound to: 127.22.193.2:36389
I20260925 03:03:49.615958 24572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.193.2:36389 every 8 connection(s)
I20260925 03:03:49.620527 23300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 03:03:49.628332 24576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:03:49.629249 24577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:03:49.634056 24579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 03:03:49.635242 23300 server_base.cc:1034] running on GCE node
I20260925 03:03:49.636425 23300 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20260925 03:03:49.636511 24573 heartbeater.cc:347] Connected to a master server at 127.22.193.62:40353
W20260925 03:03:49.636797 23300 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260925 03:03:49.636979 24573 heartbeater.cc:464] Registering TS with master...
I20260925 03:03:49.637007 23300 hybrid_clock.cc:648] HybridClock initialized: now 1790305429636991 us; error 0 us; skew 500 ppm
I20260925 03:03:49.637867 23300 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 03:03:49.637939 24573 heartbeater.cc:511] Master 127.22.193.62:40353 requested a full tablet report, sending...
I20260925 03:03:49.640125 24371 ts_manager.cc:194] Registered new tserver with Master: a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:03:49.640877 23300 webserver.cc:460] Webserver started at http://127.22.193.3:43497/ using document root <none> and password file <none>
I20260925 03:03:49.641384 23300 fs_manager.cc:359] Metadata directory not provided
I20260925 03:03:49.641570 23300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 03:03:49.641849 23300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 03:03:49.641812 24371 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39374
I20260925 03:03:49.643814 23300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklAj8hX/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790305368086980-23300-0/minicluster-data/ts-2-root/instance:
uuid: "eeab93abfdb141a895002ec9f312296c"
format_stamp: "Formatted at 2026-09-25 03:03:49 on dist-test-slave-cswq"
I20260925 03:03:49.649384 23300 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 03:03:49.652892 24584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 03:03:49.653731 23300 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 03:03:49.654011 23300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklAj8hX/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790305368086980-23300-0/minicluster-data/ts-2-root
uuid: "eeab93abfdb141a895002ec9f312296c"
format_stamp: "Formatted at 2026-09-25 03:03:49 on dist-test-slave-cswq"
I20260925 03:03:49.654284 23300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklAj8hX/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790305368086980-23300-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-tasklAj8hX/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790305368086980-23300-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-tasklAj8hX/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790305368086980-23300-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 03:03:49.666512 23300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 03:03:49.667704 23300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 03:03:49.669229 23300 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 03:03:49.671600 23300 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 03:03:49.671808 23300 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 03:03:49.672016 23300 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 03:03:49.672140 23300 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 03:03:49.713649 23300 rpc_server.cc:307] RPC server started. Bound to: 127.22.193.3:35105
I20260925 03:03:49.713742 24646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.193.3:35105 every 8 connection(s)
I20260925 03:03:49.728513 24647 heartbeater.cc:347] Connected to a master server at 127.22.193.62:40353
I20260925 03:03:49.728966 24647 heartbeater.cc:464] Registering TS with master...
I20260925 03:03:49.729619 24647 heartbeater.cc:511] Master 127.22.193.62:40353 requested a full tablet report, sending...
I20260925 03:03:49.731547 24371 ts_manager.cc:194] Registered new tserver with Master: eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:49.732586 23300 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.01593456s
I20260925 03:03:49.733467 24371 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39382
I20260925 03:03:49.735235 23300 test_util.cc:268] Using random seed: -147994836
I20260925 03:03:49.761224 24371 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39398:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}""\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 03:03:49.767788 24371 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 03:03:49.971204 24612 tablet_service.cc:1467] Processing CreateTablet for tablet 04e068a0f80941dc9138cee063f53bab (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260925 03:03:49.972625 24612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04e068a0f80941dc9138cee063f53bab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:49.973304 24611 tablet_service.cc:1467] Processing CreateTablet for tablet f68804eb5a244d87bb395d50683c6012 (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260925 03:03:49.974656 24611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f68804eb5a244d87bb395d50683c6012. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:49.977875 24609 tablet_service.cc:1467] Processing CreateTablet for tablet f22ddd7d1c444dc58f45a1fb0b3e02bc (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260925 03:03:49.981950 24607 tablet_service.cc:1467] Processing CreateTablet for tablet b1e85248d3b04b979dd080caeeb02fdb (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260925 03:03:49.984239 24463 tablet_service.cc:1467] Processing CreateTablet for tablet 04e068a0f80941dc9138cee063f53bab (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260925 03:03:49.984625 24462 tablet_service.cc:1467] Processing CreateTablet for tablet f68804eb5a244d87bb395d50683c6012 (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260925 03:03:49.985602 24463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04e068a0f80941dc9138cee063f53bab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:49.986721 24461 tablet_service.cc:1467] Processing CreateTablet for tablet edcf6e6a422d4d198d4fbb0f7d3d845d (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260925 03:03:49.988135 24461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edcf6e6a422d4d198d4fbb0f7d3d845d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:49.989881 24460 tablet_service.cc:1467] Processing CreateTablet for tablet f22ddd7d1c444dc58f45a1fb0b3e02bc (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260925 03:03:49.990219 24610 tablet_service.cc:1467] Processing CreateTablet for tablet edcf6e6a422d4d198d4fbb0f7d3d845d (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260925 03:03:49.991312 24460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f22ddd7d1c444dc58f45a1fb0b3e02bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:49.993990 24537 tablet_service.cc:1467] Processing CreateTablet for tablet f68804eb5a244d87bb395d50683c6012 (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260925 03:03:49.995420 24537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f68804eb5a244d87bb395d50683c6012. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:49.995086 24529 tablet_service.cc:1467] Processing CreateTablet for tablet 293ef69caf514ac98510a68495f54eda (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260925 03:03:49.996515 24529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 293ef69caf514ac98510a68495f54eda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:49.997975 24535 tablet_service.cc:1467] Processing CreateTablet for tablet f22ddd7d1c444dc58f45a1fb0b3e02bc (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260925 03:03:49.995134 24458 tablet_service.cc:1467] Processing CreateTablet for tablet b1e85248d3b04b979dd080caeeb02fdb (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260925 03:03:50.001670 24535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f22ddd7d1c444dc58f45a1fb0b3e02bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:50.002321 24458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1e85248d3b04b979dd080caeeb02fdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:49.993889 24530 tablet_service.cc:1467] Processing CreateTablet for tablet 48ca6ccd98194cf2af7b5278d15bf4ea (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260925 03:03:50.003443 24534 tablet_service.cc:1467] Processing CreateTablet for tablet 7993f9c3e2a44f408bbf5d1e395e2306 (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260925 03:03:49.989881 24538 tablet_service.cc:1467] Processing CreateTablet for tablet 04e068a0f80941dc9138cee063f53bab (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260925 03:03:49.999599 24536 tablet_service.cc:1467] Processing CreateTablet for tablet edcf6e6a422d4d198d4fbb0f7d3d845d (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260925 03:03:49.980185 24608 tablet_service.cc:1467] Processing CreateTablet for tablet 7993f9c3e2a44f408bbf5d1e395e2306 (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260925 03:03:50.003922 24530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48ca6ccd98194cf2af7b5278d15bf4ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:50.005816 24533 tablet_service.cc:1467] Processing CreateTablet for tablet b1e85248d3b04b979dd080caeeb02fdb (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260925 03:03:50.007257 24536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edcf6e6a422d4d198d4fbb0f7d3d845d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:50.008579 24538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04e068a0f80941dc9138cee063f53bab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:49.993889 24459 tablet_service.cc:1467] Processing CreateTablet for tablet 7993f9c3e2a44f408bbf5d1e395e2306 (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260925 03:03:50.010378 24459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7993f9c3e2a44f408bbf5d1e395e2306. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:50.010852 24606 tablet_service.cc:1467] Processing CreateTablet for tablet 442d4a29b1d743599f4b6b4871e26668 (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260925 03:03:49.996268 24457 tablet_service.cc:1467] Processing CreateTablet for tablet 442d4a29b1d743599f4b6b4871e26668 (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260925 03:03:50.012456 24457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 442d4a29b1d743599f4b6b4871e26668. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:50.007216 24532 tablet_service.cc:1467] Processing CreateTablet for tablet 442d4a29b1d743599f4b6b4871e26668 (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260925 03:03:50.014070 24610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edcf6e6a422d4d198d4fbb0f7d3d845d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:50.014439 24532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 442d4a29b1d743599f4b6b4871e26668. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:50.015280 24608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7993f9c3e2a44f408bbf5d1e395e2306. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:49.992367 24531 tablet_service.cc:1467] Processing CreateTablet for tablet e92f377a79354a6ebb4eec67303c13e3 (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260925 03:03:50.016389 24607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1e85248d3b04b979dd080caeeb02fdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:50.016500 24531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e92f377a79354a6ebb4eec67303c13e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:50.017953 24462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f68804eb5a244d87bb395d50683c6012. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:50.018363 24609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f22ddd7d1c444dc58f45a1fb0b3e02bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:50.019238 24534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7993f9c3e2a44f408bbf5d1e395e2306. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:50.019510 24606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 442d4a29b1d743599f4b6b4871e26668. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:50.020598 24533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1e85248d3b04b979dd080caeeb02fdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:50.023154 24456 tablet_service.cc:1467] Processing CreateTablet for tablet e92f377a79354a6ebb4eec67303c13e3 (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260925 03:03:50.026700 24455 tablet_service.cc:1467] Processing CreateTablet for tablet 48ca6ccd98194cf2af7b5278d15bf4ea (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260925 03:03:50.028762 24456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e92f377a79354a6ebb4eec67303c13e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:50.030318 24455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48ca6ccd98194cf2af7b5278d15bf4ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:50.064961 24454 tablet_service.cc:1467] Processing CreateTablet for tablet 293ef69caf514ac98510a68495f54eda (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260925 03:03:50.066344 24454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 293ef69caf514ac98510a68495f54eda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:50.075949 24605 tablet_service.cc:1467] Processing CreateTablet for tablet e92f377a79354a6ebb4eec67303c13e3 (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260925 03:03:50.088668 24604 tablet_service.cc:1467] Processing CreateTablet for tablet 48ca6ccd98194cf2af7b5278d15bf4ea (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260925 03:03:50.090665 24603 tablet_service.cc:1467] Processing CreateTablet for tablet 293ef69caf514ac98510a68495f54eda (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260925 03:03:50.103178 24605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e92f377a79354a6ebb4eec67303c13e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:50.109515 24603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 293ef69caf514ac98510a68495f54eda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:50.112376 24604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48ca6ccd98194cf2af7b5278d15bf4ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:50.166610 24612 tablet_service.cc:1467] Processing CreateTablet for tablet 1e725b82985746b39c0cc8f671c95353 (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260925 03:03:50.171957 24612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e725b82985746b39c0cc8f671c95353. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:50.172402 24535 tablet_service.cc:1467] Processing CreateTablet for tablet 1e725b82985746b39c0cc8f671c95353 (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260925 03:03:50.177584 24535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e725b82985746b39c0cc8f671c95353. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:50.173703 24532 tablet_service.cc:1467] Processing CreateTablet for tablet 71f461a6e2ff434993b78a2d077f0abd (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260925 03:03:50.187637 24532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71f461a6e2ff434993b78a2d077f0abd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:50.244973 24663 tablet_bootstrap.cc:492] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Bootstrap starting.
I20260925 03:03:50.245645 24610 tablet_service.cc:1467] Processing CreateTablet for tablet 71f461a6e2ff434993b78a2d077f0abd (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260925 03:03:50.246961 24610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71f461a6e2ff434993b78a2d077f0abd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:50.247362 24457 tablet_service.cc:1467] Processing CreateTablet for tablet 0b5df303cd6a4d59aca5c18bbbff24ed (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260925 03:03:50.269053 24537 tablet_service.cc:1467] Processing CreateTablet for tablet 04ac285175464fbda67211d9574db415 (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260925 03:03:50.270606 24537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04ac285175464fbda67211d9574db415. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:50.243649 24534 tablet_service.cc:1467] Processing CreateTablet for tablet 0e54ca1757ee4e42956a9cebbeca4f62 (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260925 03:03:50.277731 24534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e54ca1757ee4e42956a9cebbeca4f62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:50.283192 24536 tablet_service.cc:1467] Processing CreateTablet for tablet 06f4a7eaf6534e4986c9fd35911038e4 (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260925 03:03:50.284511 24662 tablet_bootstrap.cc:492] T edcf6e6a422d4d198d4fbb0f7d3d845d P eeab93abfdb141a895002ec9f312296c: Bootstrap starting.
I20260925 03:03:50.294973 24663 tablet_bootstrap.cc:654] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:50.258327 24533 tablet_service.cc:1467] Processing CreateTablet for tablet c85f5d1d3f004cf5817bb594fd8e6469 (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260925 03:03:50.305104 24536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06f4a7eaf6534e4986c9fd35911038e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:50.305913 24460 tablet_service.cc:1467] Processing CreateTablet for tablet 1e725b82985746b39c0cc8f671c95353 (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260925 03:03:50.315497 24611 tablet_service.cc:1467] Processing CreateTablet for tablet 04ac285175464fbda67211d9574db415 (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260925 03:03:50.308199 24459 tablet_service.cc:1467] Processing CreateTablet for tablet 71f461a6e2ff434993b78a2d077f0abd (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260925 03:03:50.333369 24611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04ac285175464fbda67211d9574db415. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:50.311708 24664 tablet_bootstrap.cc:492] T e92f377a79354a6ebb4eec67303c13e3 P 869b0a8575b54d08ae2817fad5dd42d5: Bootstrap starting.
I20260925 03:03:50.312867 24609 tablet_service.cc:1467] Processing CreateTablet for tablet 0e54ca1757ee4e42956a9cebbeca4f62 (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260925 03:03:50.337369 24609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e54ca1757ee4e42956a9cebbeca4f62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:50.316023 24606 tablet_service.cc:1467] Processing CreateTablet for tablet c85f5d1d3f004cf5817bb594fd8e6469 (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260925 03:03:50.318020 24608 tablet_service.cc:1467] Processing CreateTablet for tablet 06f4a7eaf6534e4986c9fd35911038e4 (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260925 03:03:50.319697 24610 tablet_service.cc:1467] Processing CreateTablet for tablet 29637f5a4a504ce6ba29033ccd0cd716 (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260925 03:03:50.321000 24605 tablet_service.cc:1467] Processing CreateTablet for tablet 0b5df303cd6a4d59aca5c18bbbff24ed (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260925 03:03:50.342602 24610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29637f5a4a504ce6ba29033ccd0cd716. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:50.326684 24538 tablet_service.cc:1467] Processing CreateTablet for tablet 4e5332a5d329493991dff994e70ea92c (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260925 03:03:50.329005 24533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c85f5d1d3f004cf5817bb594fd8e6469. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:50.344832 24457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b5df303cd6a4d59aca5c18bbbff24ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:50.346114 24459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71f461a6e2ff434993b78a2d077f0abd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:50.348990 24460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e725b82985746b39c0cc8f671c95353. 1 dirs total, 0 dirs full, 0 dirs failed
W20260925 03:03:50.351466 24418 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260925 03:03:50.352656 24605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b5df303cd6a4d59aca5c18bbbff24ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:50.352651 24529 tablet_service.cc:1467] Processing CreateTablet for tablet 0b5df303cd6a4d59aca5c18bbbff24ed (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260925 03:03:50.353971 24529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b5df303cd6a4d59aca5c18bbbff24ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:50.307880 24530 tablet_service.cc:1467] Processing CreateTablet for tablet 29637f5a4a504ce6ba29033ccd0cd716 (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260925 03:03:50.355093 24530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29637f5a4a504ce6ba29033ccd0cd716. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:50.329813 24456 tablet_service.cc:1467] Processing CreateTablet for tablet c85f5d1d3f004cf5817bb594fd8e6469 (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260925 03:03:50.357092 24538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e5332a5d329493991dff994e70ea92c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:50.357605 24456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c85f5d1d3f004cf5817bb594fd8e6469. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:50.358249 24604 tablet_service.cc:1467] Processing CreateTablet for tablet c5aec38e66a947c681e6c2a72b05e6ca (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260925 03:03:50.359277 24608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06f4a7eaf6534e4986c9fd35911038e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:50.362318 24454 tablet_service.cc:1467] Processing CreateTablet for tablet b627ecb4a5384ef392aa695e06599c39 (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260925 03:03:50.363547 24454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b627ecb4a5384ef392aa695e06599c39. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:50.363323 24455 tablet_service.cc:1467] Processing CreateTablet for tablet 0e54ca1757ee4e42956a9cebbeca4f62 (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260925 03:03:50.365140 24604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5aec38e66a947c681e6c2a72b05e6ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:50.341276 24461 tablet_service.cc:1467] Processing CreateTablet for tablet 04ac285175464fbda67211d9574db415 (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260925 03:03:50.340260 24607 tablet_service.cc:1467] Processing CreateTablet for tablet b627ecb4a5384ef392aa695e06599c39 (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260925 03:03:50.367964 24458 tablet_service.cc:1467] Processing CreateTablet for tablet 06f4a7eaf6534e4986c9fd35911038e4 (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260925 03:03:50.370147 24606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c85f5d1d3f004cf5817bb594fd8e6469. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:50.371474 24607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b627ecb4a5384ef392aa695e06599c39. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:50.373773 24463 tablet_service.cc:1467] Processing CreateTablet for tablet 29637f5a4a504ce6ba29033ccd0cd716 (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260925 03:03:50.377230 24455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e54ca1757ee4e42956a9cebbeca4f62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:50.378281 24612 tablet_service.cc:1467] Processing CreateTablet for tablet 60d45b7d467d4e1ab92367f6d7d52b15 (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260925 03:03:50.380712 24461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04ac285175464fbda67211d9574db415. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:50.383008 24458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06f4a7eaf6534e4986c9fd35911038e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:50.387048 24463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29637f5a4a504ce6ba29033ccd0cd716. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:50.390183 24537 tablet_service.cc:1467] Processing CreateTablet for tablet b627ecb4a5384ef392aa695e06599c39 (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260925 03:03:50.391659 24537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b627ecb4a5384ef392aa695e06599c39. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:50.400960 24535 tablet_service.cc:1467] Processing CreateTablet for tablet c5aec38e66a947c681e6c2a72b05e6ca (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260925 03:03:50.402381 24535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5aec38e66a947c681e6c2a72b05e6ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:50.427229 24612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60d45b7d467d4e1ab92367f6d7d52b15. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:50.405709 24662 tablet_bootstrap.cc:654] T edcf6e6a422d4d198d4fbb0f7d3d845d P eeab93abfdb141a895002ec9f312296c: Neither blocks nor log segments found. Creating new log.
W20260925 03:03:50.412560 24356 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 52964us
I20260925 03:03:50.412762 24534 tablet_service.cc:1467] Processing CreateTablet for tablet 8ac466793d094325ace224708b2158ed (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260925 03:03:50.417995 24603 tablet_service.cc:1467] Processing CreateTablet for tablet 8ac466793d094325ace224708b2158ed (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260925 03:03:50.418670 24663 tablet_bootstrap.cc:492] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5: No bootstrap required, opened a new log
I20260925 03:03:50.419004 24532 tablet_service.cc:1467] Processing CreateTablet for tablet 43366a0a2f9d4344bf4e37ffd290ef8a (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260925 03:03:50.431883 24664 tablet_bootstrap.cc:654] T e92f377a79354a6ebb4eec67303c13e3 P 869b0a8575b54d08ae2817fad5dd42d5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:50.437697 24603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ac466793d094325ace224708b2158ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:50.440420 24531 tablet_service.cc:1467] Processing CreateTablet for tablet 60d45b7d467d4e1ab92367f6d7d52b15 (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260925 03:03:50.443192 24462 tablet_service.cc:1467] Processing CreateTablet for tablet c5aec38e66a947c681e6c2a72b05e6ca (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260925 03:03:50.444568 24462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5aec38e66a947c681e6c2a72b05e6ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:50.434800 24663 ts_tablet_manager.cc:1397] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent bootstrapping tablet: real 0.191s user 0.013s sys 0.006s
I20260925 03:03:50.435745 24534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ac466793d094325ace224708b2158ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:50.456099 24531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60d45b7d467d4e1ab92367f6d7d52b15. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:50.456792 24532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43366a0a2f9d4344bf4e37ffd290ef8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:50.476969 24663 raft_consensus.cc:348] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:50.477768 24663 raft_consensus.cc:374] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:50.478046 24663 raft_consensus.cc:729] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Initialized, Role: FOLLOWER
I20260925 03:03:50.478726 24663 consensus_queue.cc:260] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:50.492038 24530 tablet_service.cc:1467] Processing CreateTablet for tablet 9e8af2641e124f9589c5ccfdac785f63 (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260925 03:03:50.497994 24530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e8af2641e124f9589c5ccfdac785f63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:50.518905 24610 tablet_service.cc:1467] Processing CreateTablet for tablet 43366a0a2f9d4344bf4e37ffd290ef8a (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260925 03:03:50.520402 24610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43366a0a2f9d4344bf4e37ffd290ef8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:50.530509 24605 tablet_service.cc:1467] Processing CreateTablet for tablet 4e5332a5d329493991dff994e70ea92c (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260925 03:03:50.536151 24662 tablet_bootstrap.cc:492] T edcf6e6a422d4d198d4fbb0f7d3d845d P eeab93abfdb141a895002ec9f312296c: No bootstrap required, opened a new log
I20260925 03:03:50.538026 24498 heartbeater.cc:503] Master 127.22.193.62:40353 was elected leader, sending a full tablet report...
I20260925 03:03:50.544318 24460 tablet_service.cc:1467] Processing CreateTablet for tablet 60d45b7d467d4e1ab92367f6d7d52b15 (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260925 03:03:50.551414 24663 ts_tablet_manager.cc:1428] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent starting tablet: real 0.102s user 0.031s sys 0.034s
I20260925 03:03:50.552301 24573 heartbeater.cc:503] Master 127.22.193.62:40353 was elected leader, sending a full tablet report...
I20260925 03:03:50.552515 24663 tablet_bootstrap.cc:492] T f22ddd7d1c444dc58f45a1fb0b3e02bc P a04a37fc4aa7460cbfdbeeb4e451e1f5: Bootstrap starting.
I20260925 03:03:50.555972 24460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60d45b7d467d4e1ab92367f6d7d52b15. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:50.558868 24609 tablet_service.cc:1467] Processing CreateTablet for tablet 9e8af2641e124f9589c5ccfdac785f63 (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260925 03:03:50.568048 24533 tablet_service.cc:1467] Processing CreateTablet for tablet 74443a0a1dc34d88a2f42e3b722db405 (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260925 03:03:50.564631 24604 tablet_service.cc:1467] Processing CreateTablet for tablet abd5b8dc1b7e49e3aa9af900d73ff0e7 (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260925 03:03:50.573060 24605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e5332a5d329493991dff994e70ea92c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:50.609431 24533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74443a0a1dc34d88a2f42e3b722db405. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:50.576574 24663 tablet_bootstrap.cc:654] T f22ddd7d1c444dc58f45a1fb0b3e02bc P a04a37fc4aa7460cbfdbeeb4e451e1f5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:50.611193 24612 tablet_service.cc:1467] Processing CreateTablet for tablet a7aa47c48404465b94a36fd819fe3c7a (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260925 03:03:50.563782 24536 tablet_service.cc:1467] Processing CreateTablet for tablet abd5b8dc1b7e49e3aa9af900d73ff0e7 (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260925 03:03:50.613590 24458 tablet_service.cc:1467] Processing CreateTablet for tablet 74443a0a1dc34d88a2f42e3b722db405 (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260925 03:03:50.614512 24536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abd5b8dc1b7e49e3aa9af900d73ff0e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:50.614868 24458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74443a0a1dc34d88a2f42e3b722db405. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:50.615608 24534 tablet_service.cc:1467] Processing CreateTablet for tablet b4ebd2b043024a548005c8b6b00b0449 (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260925 03:03:50.585204 24537 tablet_service.cc:1467] Processing CreateTablet for tablet a4abe0ea757e4886be4945c5001e529a (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260925 03:03:50.592504 24662 ts_tablet_manager.cc:1397] T edcf6e6a422d4d198d4fbb0f7d3d845d P eeab93abfdb141a895002ec9f312296c: Time spent bootstrapping tablet: real 0.308s user 0.019s sys 0.005s
I20260925 03:03:50.621551 24454 tablet_service.cc:1467] Processing CreateTablet for tablet 5278a60f97da4e10aecd50a8d9a9a43c (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260925 03:03:50.592351 24460 tablet_service.cc:1467] Processing CreateTablet for tablet 43366a0a2f9d4344bf4e37ffd290ef8a (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260925 03:03:50.621455 24537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4abe0ea757e4886be4945c5001e529a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:50.582558 24457 tablet_service.cc:1467] Processing CreateTablet for tablet 8ac466793d094325ace224708b2158ed (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260925 03:03:50.602272 24603 tablet_service.cc:1467] Processing CreateTablet for tablet 74443a0a1dc34d88a2f42e3b722db405 (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260925 03:03:50.626255 24538 tablet_service.cc:1467] Processing CreateTablet for tablet a7aa47c48404465b94a36fd819fe3c7a (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260925 03:03:50.627399 24538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7aa47c48404465b94a36fd819fe3c7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:50.595306 24529 tablet_service.cc:1467] Processing CreateTablet for tablet 5278a60f97da4e10aecd50a8d9a9a43c (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260925 03:03:50.604063 24607 tablet_service.cc:1467] Processing CreateTablet for tablet 5278a60f97da4e10aecd50a8d9a9a43c (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260925 03:03:50.604672 24611 tablet_service.cc:1467] Processing CreateTablet for tablet b4ebd2b043024a548005c8b6b00b0449 (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260925 03:03:50.607506 24459 tablet_service.cc:1467] Processing CreateTablet for tablet abd5b8dc1b7e49e3aa9af900d73ff0e7 (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260925 03:03:50.607692 24606 tablet_service.cc:1467] Processing CreateTablet for tablet b030d84c670b48d5bff337ccc0c77626 (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260925 03:03:50.633165 24662 raft_consensus.cc:348] T edcf6e6a422d4d198d4fbb0f7d3d845d P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:50.634231 24662 raft_consensus.cc:374] T edcf6e6a422d4d198d4fbb0f7d3d845d P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:50.634670 24662 raft_consensus.cc:729] T edcf6e6a422d4d198d4fbb0f7d3d845d P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Initialized, Role: FOLLOWER
I20260925 03:03:50.617686 24461 tablet_service.cc:1467] Processing CreateTablet for tablet a4abe0ea757e4886be4945c5001e529a (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260925 03:03:50.619355 24664 tablet_bootstrap.cc:492] T e92f377a79354a6ebb4eec67303c13e3 P 869b0a8575b54d08ae2817fad5dd42d5: No bootstrap required, opened a new log
I20260925 03:03:50.631104 24529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5278a60f97da4e10aecd50a8d9a9a43c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:50.639433 24664 ts_tablet_manager.cc:1397] T e92f377a79354a6ebb4eec67303c13e3 P 869b0a8575b54d08ae2817fad5dd42d5: Time spent bootstrapping tablet: real 0.328s user 0.016s sys 0.009s
I20260925 03:03:50.599836 24462 tablet_service.cc:1467] Processing CreateTablet for tablet 9e8af2641e124f9589c5ccfdac785f63 (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260925 03:03:50.592355 24456 tablet_service.cc:1467] Processing CreateTablet for tablet 4e5332a5d329493991dff994e70ea92c (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260925 03:03:50.622725 24454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5278a60f97da4e10aecd50a8d9a9a43c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:50.619740 24612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7aa47c48404465b94a36fd819fe3c7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:50.602387 24608 tablet_service.cc:1467] Processing CreateTablet for tablet a4abe0ea757e4886be4945c5001e529a (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260925 03:03:50.642239 24457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ac466793d094325ace224708b2158ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:50.624747 24455 tablet_service.cc:1467] Processing CreateTablet for tablet b4ebd2b043024a548005c8b6b00b0449 (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260925 03:03:50.643071 24456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e5332a5d329493991dff994e70ea92c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:50.643745 24462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e8af2641e124f9589c5ccfdac785f63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:50.648274 24460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43366a0a2f9d4344bf4e37ffd290ef8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:50.650193 24606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b030d84c670b48d5bff337ccc0c77626. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:50.651502 24603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74443a0a1dc34d88a2f42e3b722db405. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:50.654527 24607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5278a60f97da4e10aecd50a8d9a9a43c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:50.655886 24455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4ebd2b043024a548005c8b6b00b0449. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:50.656980 24604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abd5b8dc1b7e49e3aa9af900d73ff0e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:50.660475 24459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abd5b8dc1b7e49e3aa9af900d73ff0e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:50.661011 24609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e8af2641e124f9589c5ccfdac785f63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:50.664218 24461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4abe0ea757e4886be4945c5001e529a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:50.663810 24664 raft_consensus.cc:348] T e92f377a79354a6ebb4eec67303c13e3 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:50.664752 24664 raft_consensus.cc:374] T e92f377a79354a6ebb4eec67303c13e3 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:50.665098 24664 raft_consensus.cc:729] T e92f377a79354a6ebb4eec67303c13e3 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Initialized, Role: FOLLOWER
I20260925 03:03:50.665899 24664 consensus_queue.cc:260] T e92f377a79354a6ebb4eec67303c13e3 P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:50.669045 24611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4ebd2b043024a548005c8b6b00b0449. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:50.689031 24608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4abe0ea757e4886be4945c5001e529a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:50.670027 24534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4ebd2b043024a548005c8b6b00b0449. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:50.636749 24662 consensus_queue.cc:260] T edcf6e6a422d4d198d4fbb0f7d3d845d P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:50.669968 24535 tablet_service.cc:1467] Processing CreateTablet for tablet 367277daf8d04778b3fa4ed94d71580d (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260925 03:03:50.670850 24531 tablet_service.cc:1467] Processing CreateTablet for tablet b030d84c670b48d5bff337ccc0c77626 (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260925 03:03:50.674638 24610 tablet_service.cc:1467] Processing CreateTablet for tablet 3d89396576884f7c8901dba54f783947 (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260925 03:03:50.694535 24610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d89396576884f7c8901dba54f783947. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:50.674752 24532 tablet_service.cc:1467] Processing CreateTablet for tablet 37a157996e944db2906110d87ecb0c54 (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260925 03:03:50.676354 24463 tablet_service.cc:1467] Processing CreateTablet for tablet b030d84c670b48d5bff337ccc0c77626 (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260925 03:03:50.697906 24463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b030d84c670b48d5bff337ccc0c77626. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:50.641171 24530 tablet_service.cc:1467] Processing CreateTablet for tablet 3d89396576884f7c8901dba54f783947 (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260925 03:03:50.715758 24533 tablet_service.cc:1467] Processing CreateTablet for tablet 1414dcfe53f64a78ae413531eb8ec117 (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260925 03:03:50.694535 24531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b030d84c670b48d5bff337ccc0c77626. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:50.718792 24532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37a157996e944db2906110d87ecb0c54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:50.719646 24530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d89396576884f7c8901dba54f783947. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:50.727386 24456 tablet_service.cc:1467] Processing CreateTablet for tablet a7aa47c48404465b94a36fd819fe3c7a (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260925 03:03:50.730664 24536 tablet_service.cc:1467] Processing CreateTablet for tablet 2c23337a7cb74184a0e57b7095cb6a11 (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260925 03:03:50.733999 24535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 367277daf8d04778b3fa4ed94d71580d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:50.734490 24460 tablet_service.cc:1467] Processing CreateTablet for tablet 3d89396576884f7c8901dba54f783947 (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260925 03:03:50.728564 24605 tablet_service.cc:1467] Processing CreateTablet for tablet 367277daf8d04778b3fa4ed94d71580d (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260925 03:03:50.738427 24647 heartbeater.cc:503] Master 127.22.193.62:40353 was elected leader, sending a full tablet report...
I20260925 03:03:50.738899 24533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1414dcfe53f64a78ae413531eb8ec117. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:50.749913 24456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7aa47c48404465b94a36fd819fe3c7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:50.750859 24605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 367277daf8d04778b3fa4ed94d71580d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:50.753073 24460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d89396576884f7c8901dba54f783947. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:50.754185 24663 tablet_bootstrap.cc:492] T f22ddd7d1c444dc58f45a1fb0b3e02bc P a04a37fc4aa7460cbfdbeeb4e451e1f5: No bootstrap required, opened a new log
I20260925 03:03:50.754730 24663 ts_tablet_manager.cc:1397] T f22ddd7d1c444dc58f45a1fb0b3e02bc P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent bootstrapping tablet: real 0.202s user 0.018s sys 0.006s
I20260925 03:03:50.755133 24536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c23337a7cb74184a0e57b7095cb6a11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:50.819591 24538 tablet_service.cc:1467] Processing CreateTablet for tablet 7e49248643df4131a1c42e74244e05c2 (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
W20260925 03:03:50.822017 24356 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 106850us
I20260925 03:03:50.826516 24663 raft_consensus.cc:348] T f22ddd7d1c444dc58f45a1fb0b3e02bc P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:50.827315 24663 raft_consensus.cc:374] T f22ddd7d1c444dc58f45a1fb0b3e02bc P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:50.827617 24663 raft_consensus.cc:729] T f22ddd7d1c444dc58f45a1fb0b3e02bc P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Initialized, Role: FOLLOWER
W20260925 03:03:50.829030 24358 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 113376us
I20260925 03:03:50.830130 24664 ts_tablet_manager.cc:1428] T e92f377a79354a6ebb4eec67303c13e3 P 869b0a8575b54d08ae2817fad5dd42d5: Time spent starting tablet: real 0.190s user 0.006s sys 0.009s
I20260925 03:03:50.832748 24459 tablet_service.cc:1467] Processing CreateTablet for tablet 367277daf8d04778b3fa4ed94d71580d (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260925 03:03:50.833242 24667 raft_consensus.cc:482] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:50.833736 24667 raft_consensus.cc:504] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:50.836076 24667 leader_election.cc:288] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:50.838744 24454 tablet_service.cc:1467] Processing CreateTablet for tablet 37a157996e944db2906110d87ecb0c54 (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260925 03:03:50.846159 24662 ts_tablet_manager.cc:1428] T edcf6e6a422d4d198d4fbb0f7d3d845d P eeab93abfdb141a895002ec9f312296c: Time spent starting tablet: real 0.224s user 0.008s sys 0.011s
I20260925 03:03:50.847450 24662 tablet_bootstrap.cc:492] T 04e068a0f80941dc9138cee063f53bab P eeab93abfdb141a895002ec9f312296c: Bootstrap starting.
I20260925 03:03:50.855078 24670 raft_consensus.cc:482] T edcf6e6a422d4d198d4fbb0f7d3d845d P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:50.855682 24670 raft_consensus.cc:504] T edcf6e6a422d4d198d4fbb0f7d3d845d P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:50.858363 24670 leader_election.cc:288] T edcf6e6a422d4d198d4fbb0f7d3d845d P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:03:50.885695 24538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e49248643df4131a1c42e74244e05c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:50.859670 24463 tablet_service.cc:1467] Processing CreateTablet for tablet 2c23337a7cb74184a0e57b7095cb6a11 (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260925 03:03:50.861244 24537 tablet_service.cc:1467] Processing CreateTablet for tablet 9f1fac028a144058b92331c0d9b44ebf (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260925 03:03:50.889467 24537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f1fac028a144058b92331c0d9b44ebf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:50.862686 24664 tablet_bootstrap.cc:492] T 442d4a29b1d743599f4b6b4871e26668 P 869b0a8575b54d08ae2817fad5dd42d5: Bootstrap starting.
I20260925 03:03:50.851037 24458 tablet_service.cc:1467] Processing CreateTablet for tablet 1414dcfe53f64a78ae413531eb8ec117 (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260925 03:03:50.894572 24603 tablet_service.cc:1467] Processing CreateTablet for tablet 1414dcfe53f64a78ae413531eb8ec117 (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260925 03:03:50.874778 24663 consensus_queue.cc:260] T f22ddd7d1c444dc58f45a1fb0b3e02bc P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:50.902550 24663 ts_tablet_manager.cc:1428] T f22ddd7d1c444dc58f45a1fb0b3e02bc P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent starting tablet: real 0.147s user 0.002s sys 0.007s
I20260925 03:03:50.903868 24663 tablet_bootstrap.cc:492] T 442d4a29b1d743599f4b6b4871e26668 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Bootstrap starting.
I20260925 03:03:50.877271 24531 tablet_service.cc:1467] Processing CreateTablet for tablet 0be4e0aa984f4f2b81a58e382e434ad5 (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260925 03:03:50.910109 24606 tablet_service.cc:1467] Processing CreateTablet for tablet 7e49248643df4131a1c42e74244e05c2 (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260925 03:03:50.911060 24531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0be4e0aa984f4f2b81a58e382e434ad5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:50.882754 24457 tablet_service.cc:1467] Processing CreateTablet for tablet 9f1fac028a144058b92331c0d9b44ebf (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260925 03:03:50.895887 24529 tablet_service.cc:1467] Processing CreateTablet for tablet 550118bc1a0641ce9803bdbf43f2069f (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260925 03:03:50.914080 24529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 550118bc1a0641ce9803bdbf43f2069f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:50.897135 24604 tablet_service.cc:1467] Processing CreateTablet for tablet 2c23337a7cb74184a0e57b7095cb6a11 (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260925 03:03:50.834070 24459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 367277daf8d04778b3fa4ed94d71580d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:50.898906 24662 tablet_bootstrap.cc:654] T 04e068a0f80941dc9138cee063f53bab P eeab93abfdb141a895002ec9f312296c: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:50.900095 24534 tablet_service.cc:1467] Processing CreateTablet for tablet 15861cd28e354598a3c6bfc4337c34bb (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260925 03:03:50.872257 24609 tablet_service.cc:1467] Processing CreateTablet for tablet 37a157996e944db2906110d87ecb0c54 (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260925 03:03:50.868325 24455 tablet_service.cc:1467] Processing CreateTablet for tablet 7e49248643df4131a1c42e74244e05c2 (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260925 03:03:50.910109 24456 tablet_service.cc:1467] Processing CreateTablet for tablet 550118bc1a0641ce9803bdbf43f2069f (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260925 03:03:50.922160 24607 tablet_service.cc:1467] Processing CreateTablet for tablet 9f1fac028a144058b92331c0d9b44ebf (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260925 03:03:50.924969 24671 raft_consensus.cc:482] T e92f377a79354a6ebb4eec67303c13e3 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:50.941243 24457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f1fac028a144058b92331c0d9b44ebf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:50.940923 24606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e49248643df4131a1c42e74244e05c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:50.943866 24610 tablet_service.cc:1467] Processing CreateTablet for tablet 550118bc1a0641ce9803bdbf43f2069f (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260925 03:03:50.929172 24671 raft_consensus.cc:504] T e92f377a79354a6ebb4eec67303c13e3 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:50.953822 24671 leader_election.cc:288] T e92f377a79354a6ebb4eec67303c13e3 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:50.936681 24461 tablet_service.cc:1467] Processing CreateTablet for tablet c88cbd39e01f44628a094af516be2833 (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260925 03:03:50.956920 24534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15861cd28e354598a3c6bfc4337c34bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:50.968940 24603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1414dcfe53f64a78ae413531eb8ec117. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:50.983937 24609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37a157996e944db2906110d87ecb0c54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:50.982277 24454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37a157996e944db2906110d87ecb0c54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:50.989114 24663 tablet_bootstrap.cc:654] T 442d4a29b1d743599f4b6b4871e26668 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:50.997681 24605 tablet_service.cc:1467] Processing CreateTablet for tablet 15861cd28e354598a3c6bfc4337c34bb (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260925 03:03:51.016727 24608 tablet_service.cc:1467] Processing CreateTablet for tablet 0be4e0aa984f4f2b81a58e382e434ad5 (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260925 03:03:51.019024 24462 tablet_service.cc:1467] Processing CreateTablet for tablet 21c43af5e6844fdaacc8c625fa01fd8d (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260925 03:03:51.013156 24611 tablet_service.cc:1467] Processing CreateTablet for tablet 21c43af5e6844fdaacc8c625fa01fd8d (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260925 03:03:51.023469 24667 raft_consensus.cc:482] T f22ddd7d1c444dc58f45a1fb0b3e02bc P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:51.023994 24667 raft_consensus.cc:504] T f22ddd7d1c444dc58f45a1fb0b3e02bc P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:51.026129 24667 leader_election.cc:288] T f22ddd7d1c444dc58f45a1fb0b3e02bc P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:51.011610 24612 tablet_service.cc:1467] Processing CreateTablet for tablet 8e5819c15eac4544a57e5ee6c9d4f117 (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260925 03:03:51.003881 24604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c23337a7cb74184a0e57b7095cb6a11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:51.028888 24607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f1fac028a144058b92331c0d9b44ebf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:51.035851 24455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e49248643df4131a1c42e74244e05c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:51.038139 24463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c23337a7cb74184a0e57b7095cb6a11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:51.038956 24605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15861cd28e354598a3c6bfc4337c34bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:51.042348 24458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1414dcfe53f64a78ae413531eb8ec117. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:51.043618 24611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21c43af5e6844fdaacc8c625fa01fd8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:51.048245 24533 tablet_service.cc:1467] Processing CreateTablet for tablet 11bb46bce3564841b7674e6a693bd2e6 (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260925 03:03:51.049676 24533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11bb46bce3564841b7674e6a693bd2e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:51.051349 24610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 550118bc1a0641ce9803bdbf43f2069f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:51.055418 24456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 550118bc1a0641ce9803bdbf43f2069f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:51.057091 24612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e5819c15eac4544a57e5ee6c9d4f117. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:51.057039 24461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c88cbd39e01f44628a094af516be2833. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:51.062906 24462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21c43af5e6844fdaacc8c625fa01fd8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:51.070971 24608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0be4e0aa984f4f2b81a58e382e434ad5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:51.076572 24531 tablet_service.cc:1467] Processing CreateTablet for tablet ac9357d144aa49789faaf31fe5e6bf89 (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260925 03:03:51.077989 24531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac9357d144aa49789faaf31fe5e6bf89. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:51.084208 24664 tablet_bootstrap.cc:654] T 442d4a29b1d743599f4b6b4871e26668 P 869b0a8575b54d08ae2817fad5dd42d5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:51.007429 24530 tablet_service.cc:1467] Processing CreateTablet for tablet 8e5819c15eac4544a57e5ee6c9d4f117 (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260925 03:03:51.089956 24530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e5819c15eac4544a57e5ee6c9d4f117. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:51.009078 24536 tablet_service.cc:1467] Processing CreateTablet for tablet 21c43af5e6844fdaacc8c625fa01fd8d (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260925 03:03:51.092222 24532 tablet_service.cc:1467] Processing CreateTablet for tablet c88cbd39e01f44628a094af516be2833 (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260925 03:03:51.105712 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f68804eb5a244d87bb395d50683c6012" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:03:51.106540 24536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21c43af5e6844fdaacc8c625fa01fd8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:51.106755 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f68804eb5a244d87bb395d50683c6012" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:03:51.108469 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f22ddd7d1c444dc58f45a1fb0b3e02bc" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
W20260925 03:03:51.110320 24511 leader_election.cc:341] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105): Illegal state: must be running to vote when last-logged opid is not known
W20260925 03:03:51.111025 24514 leader_election.cc:341] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:51.111444 24514 leader_election.cc:302] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:03:51.112267 24667 raft_consensus.cc:2726] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260925 03:03:51.112655 24514 leader_election.cc:341] T f22ddd7d1c444dc58f45a1fb0b3e02bc P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:51.115525 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f22ddd7d1c444dc58f45a1fb0b3e02bc" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
W20260925 03:03:51.121407 24511 leader_election.cc:341] T f22ddd7d1c444dc58f45a1fb0b3e02bc P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:51.122960 24511 leader_election.cc:302] T f22ddd7d1c444dc58f45a1fb0b3e02bc P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:03:51.123842 24667 raft_consensus.cc:2726] T f22ddd7d1c444dc58f45a1fb0b3e02bc P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 03:03:51.136751 24532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c88cbd39e01f44628a094af516be2833. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:51.139467 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e92f377a79354a6ebb4eec67303c13e3" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:03:51.146559 24529 tablet_service.cc:1467] Processing CreateTablet for tablet 6c81a1b3b616492ab6c964f45af36077 (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260925 03:03:51.148780 24663 tablet_bootstrap.cc:492] T 442d4a29b1d743599f4b6b4871e26668 P a04a37fc4aa7460cbfdbeeb4e451e1f5: No bootstrap required, opened a new log
I20260925 03:03:51.149248 24663 ts_tablet_manager.cc:1397] T 442d4a29b1d743599f4b6b4871e26668 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent bootstrapping tablet: real 0.246s user 0.013s sys 0.000s
I20260925 03:03:51.148476 24534 tablet_service.cc:1467] Processing CreateTablet for tablet a9c33fd100ff44c1a3247319a324a044 (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260925 03:03:51.151908 24663 raft_consensus.cc:348] T 442d4a29b1d743599f4b6b4871e26668 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:03:51.153782 24436 leader_election.cc:341] T e92f377a79354a6ebb4eec67303c13e3 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:51.154795 24662 tablet_bootstrap.cc:492] T 04e068a0f80941dc9138cee063f53bab P eeab93abfdb141a895002ec9f312296c: No bootstrap required, opened a new log
I20260925 03:03:51.155639 24662 ts_tablet_manager.cc:1397] T 04e068a0f80941dc9138cee063f53bab P eeab93abfdb141a895002ec9f312296c: Time spent bootstrapping tablet: real 0.308s user 0.026s sys 0.025s
I20260925 03:03:51.153119 24663 raft_consensus.cc:374] T 442d4a29b1d743599f4b6b4871e26668 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:51.157481 24663 raft_consensus.cc:729] T 442d4a29b1d743599f4b6b4871e26668 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Initialized, Role: FOLLOWER
I20260925 03:03:51.159237 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcf6e6a422d4d198d4fbb0f7d3d845d" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
W20260925 03:03:51.160948 24586 leader_election.cc:341] T edcf6e6a422d4d198d4fbb0f7d3d845d P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:51.161970 24531 tablet_service.cc:1467] Processing CreateTablet for tablet 1d7691c6b475428c82d6181d693d26f3 (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260925 03:03:51.163868 24662 raft_consensus.cc:348] T 04e068a0f80941dc9138cee063f53bab P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } }
I20260925 03:03:51.164772 24662 raft_consensus.cc:374] T 04e068a0f80941dc9138cee063f53bab P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:51.165124 24662 raft_consensus.cc:729] T 04e068a0f80941dc9138cee063f53bab P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Initialized, Role: FOLLOWER
I20260925 03:03:51.165982 24662 consensus_queue.cc:260] T 04e068a0f80941dc9138cee063f53bab P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } }
I20260925 03:03:51.167779 24604 tablet_service.cc:1467] Processing CreateTablet for tablet c88cbd39e01f44628a094af516be2833 (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260925 03:03:51.169210 24604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c88cbd39e01f44628a094af516be2833. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:51.169013 24663 consensus_queue.cc:260] T 442d4a29b1d743599f4b6b4871e26668 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:51.174263 24529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c81a1b3b616492ab6c964f45af36077. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:51.179164 24670 raft_consensus.cc:482] T 04e068a0f80941dc9138cee063f53bab P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:51.179682 24670 raft_consensus.cc:504] T 04e068a0f80941dc9138cee063f53bab P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } }
I20260925 03:03:51.181850 24670 leader_election.cc:288] T 04e068a0f80941dc9138cee063f53bab P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187)
I20260925 03:03:51.182664 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e92f377a79354a6ebb4eec67303c13e3" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:03:51.184013 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e068a0f80941dc9138cee063f53bab" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
W20260925 03:03:51.184196 24437 leader_election.cc:341] T e92f377a79354a6ebb4eec67303c13e3 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:51.184680 24437 leader_election.cc:302] T e92f377a79354a6ebb4eec67303c13e3 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
W20260925 03:03:51.185626 24586 leader_election.cc:341] T 04e068a0f80941dc9138cee063f53bab P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:51.185942 24671 raft_consensus.cc:2726] T e92f377a79354a6ebb4eec67303c13e3 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 03:03:51.192950 24531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d7691c6b475428c82d6181d693d26f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:51.152482 24535 tablet_service.cc:1467] Processing CreateTablet for tablet b5f9b503b0ab43999d582015a367441e (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260925 03:03:51.196831 24538 tablet_service.cc:1467] Processing CreateTablet for tablet ceab6b0fccd54e909a64f66fe81a9a1f (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260925 03:03:51.197842 24535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5f9b503b0ab43999d582015a367441e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:51.160578 24537 tablet_service.cc:1467] Processing CreateTablet for tablet 25102ba08e4e49388f6176badcd10a25 (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260925 03:03:51.202126 24609 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:38550 (request call id 34) took 1034 ms. Trace:
I20260925 03:03:51.202548 24609 rpcz_store.cc:276] 0925 03:03:50.168214 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:03:50.821035 (+652821us) service_pool.cc:227] Handling call
0925 03:03:50.936190 (+115155us) spinlock_profiling.cc:243] Waited 1.11 ms on lock 0x7b58001d3d98. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f43733ecf11 00007f4373570851 00007f4373570702 00007f437c460b09 00007f437b39c730 00007f437b47ed21 00007f437b45c1e3 00007f437b40d869 00007f4374a0fc44 00007f4374a0fbd2 00007f4374a0fb01
0925 03:03:50.959158 (+ 22968us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0925 03:03:50.960581 (+ 1423us) ts_tablet_manager.cc:676] Creating new metadata...
0925 03:03:51.123656 (+163075us) tablet_metadata.cc:641] Metadata flushed
0925 03:03:51.201610 (+ 77954us) tablet_replica.cc:205] Creating consensus instance
0925 03:03:51.202064 (+ 454us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2946688}
I20260925 03:03:51.209544 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcf6e6a422d4d198d4fbb0f7d3d845d" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:03:51.209873 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e068a0f80941dc9138cee063f53bab" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:03:51.211076 24460 tablet_service.cc:1467] Processing CreateTablet for tablet 0be4e0aa984f4f2b81a58e382e434ad5 (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
W20260925 03:03:51.211371 24588 leader_election.cc:341] T edcf6e6a422d4d198d4fbb0f7d3d845d P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:51.211890 24588 leader_election.cc:302] T edcf6e6a422d4d198d4fbb0f7d3d845d P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:03:51.212445 24460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0be4e0aa984f4f2b81a58e382e434ad5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:51.212875 24670 raft_consensus.cc:2726] T edcf6e6a422d4d198d4fbb0f7d3d845d P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260925 03:03:51.214114 24588 leader_election.cc:341] T 04e068a0f80941dc9138cee063f53bab P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:51.214555 24588 leader_election.cc:302] T 04e068a0f80941dc9138cee063f53bab P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:03:51.215641 24670 raft_consensus.cc:2726] T 04e068a0f80941dc9138cee063f53bab P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 03:03:51.217044 24607 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:38550 (request call id 38) took 1020 ms. Trace:
I20260925 03:03:51.217450 24607 rpcz_store.cc:276] 0925 03:03:50.196584 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:03:50.921268 (+724684us) service_pool.cc:227] Handling call
0925 03:03:50.950052 (+ 28784us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0925 03:03:50.958154 (+ 8102us) ts_tablet_manager.cc:676] Creating new metadata...
0925 03:03:51.180108 (+221954us) tablet_metadata.cc:641] Metadata flushed
0925 03:03:51.203283 (+ 23175us) tablet_replica.cc:205] Creating consensus instance
0925 03:03:51.216983 (+ 13700us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:03:51.220016 24607 tablet_service.cc:1467] Processing CreateTablet for tablet a9c33fd100ff44c1a3247319a324a044 (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260925 03:03:51.221824 24454 tablet_service.cc:1467] Processing CreateTablet for tablet 15861cd28e354598a3c6bfc4337c34bb (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260925 03:03:51.222123 24607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9c33fd100ff44c1a3247319a324a044. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:51.222102 24609 tablet_service.cc:1467] Processing CreateTablet for tablet ac9357d144aa49789faaf31fe5e6bf89 (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260925 03:03:51.223093 24454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15861cd28e354598a3c6bfc4337c34bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:51.223436 24609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac9357d144aa49789faaf31fe5e6bf89. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:51.225217 24603 tablet_service.cc:1467] Processing CreateTablet for tablet 11bb46bce3564841b7674e6a693bd2e6 (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260925 03:03:51.226488 24603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11bb46bce3564841b7674e6a693bd2e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:51.229234 24610 tablet_service.cc:1467] Processing CreateTablet for tablet b5f9b503b0ab43999d582015a367441e (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260925 03:03:51.230392 24538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ceab6b0fccd54e909a64f66fe81a9a1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:51.230772 24610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5f9b503b0ab43999d582015a367441e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:51.258237 24663 ts_tablet_manager.cc:1428] T 442d4a29b1d743599f4b6b4871e26668 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent starting tablet: real 0.109s user 0.008s sys 0.000s
I20260925 03:03:51.259189 24463 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:55552 (request call id 36) took 1007 ms. Trace:
I20260925 03:03:51.259577 24463 rpcz_store.cc:276] 0925 03:03:50.251612 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:03:50.858746 (+607134us) service_pool.cc:227] Handling call
0925 03:03:50.893415 (+ 34669us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0925 03:03:50.893498 (+ 83us) ts_tablet_manager.cc:676] Creating new metadata...
0925 03:03:51.176620 (+283122us) tablet_metadata.cc:641] Metadata flushed
0925 03:03:51.258683 (+ 82063us) tablet_replica.cc:205] Creating consensus instance
0925 03:03:51.259129 (+ 446us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:03:51.250725 24456 tablet_service.cc:1467] Processing CreateTablet for tablet 8e5819c15eac4544a57e5ee6c9d4f117 (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260925 03:03:51.251889 24662 ts_tablet_manager.cc:1428] T 04e068a0f80941dc9138cee063f53bab P eeab93abfdb141a895002ec9f312296c: Time spent starting tablet: real 0.096s user 0.011s sys 0.000s
I20260925 03:03:51.252838 24459 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:55552 (request call id 33) took 1001 ms. Trace:
I20260925 03:03:51.266925 24608 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:38550 (request call id 40) took 1022 ms. Trace:
I20260925 03:03:51.263830 24537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25102ba08e4e49388f6176badcd10a25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:51.273309 24459 rpcz_store.cc:276] 0925 03:03:50.250661 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:03:50.831750 (+581089us) service_pool.cc:227] Handling call
0925 03:03:50.833569 (+ 1819us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0925 03:03:50.833668 (+ 99us) ts_tablet_manager.cc:676] Creating new metadata...
0925 03:03:51.161279 (+327611us) tablet_metadata.cc:641] Metadata flushed
0925 03:03:51.196309 (+ 35030us) tablet_replica.cc:205] Creating consensus instance
0925 03:03:51.251861 (+ 55552us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:03:51.276257 24459 tablet_service.cc:1467] Processing CreateTablet for tablet 11bb46bce3564841b7674e6a693bd2e6 (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260925 03:03:51.277624 24662 tablet_bootstrap.cc:492] T 442d4a29b1d743599f4b6b4871e26668 P eeab93abfdb141a895002ec9f312296c: Bootstrap starting.
I20260925 03:03:51.264025 24458 tablet_service.cc:1467] Processing CreateTablet for tablet ac9357d144aa49789faaf31fe5e6bf89 (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260925 03:03:51.271831 24664 tablet_bootstrap.cc:492] T 442d4a29b1d743599f4b6b4871e26668 P 869b0a8575b54d08ae2817fad5dd42d5: No bootstrap required, opened a new log
I20260925 03:03:51.280486 24463 tablet_service.cc:1467] Processing CreateTablet for tablet a9c33fd100ff44c1a3247319a324a044 (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260925 03:03:51.281188 24664 ts_tablet_manager.cc:1397] T 442d4a29b1d743599f4b6b4871e26668 P 869b0a8575b54d08ae2817fad5dd42d5: Time spent bootstrapping tablet: real 0.419s user 0.007s sys 0.035s
I20260925 03:03:51.273806 24608 rpcz_store.cc:276] 0925 03:03:50.244820 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:03:50.948370 (+703550us) service_pool.cc:227] Handling call
0925 03:03:51.017656 (+ 69286us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0925 03:03:51.017763 (+ 107us) ts_tablet_manager.cc:676] Creating new metadata...
0925 03:03:51.116671 (+ 98908us) tablet_metadata.cc:641] Metadata flushed
0925 03:03:51.266386 (+149715us) tablet_replica.cc:205] Creating consensus instance
0925 03:03:51.266866 (+ 480us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:03:51.284036 24534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9c33fd100ff44c1a3247319a324a044. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:51.258224 24606 tablet_service.cc:1467] Processing CreateTablet for tablet 25102ba08e4e49388f6176badcd10a25 (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260925 03:03:51.286875 24533 tablet_service.cc:1467] Processing CreateTablet for tablet e6b6ec010caf4689b8c90449764f5cf9 (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260925 03:03:51.288206 24608 tablet_service.cc:1467] Processing CreateTablet for tablet 1d7691c6b475428c82d6181d693d26f3 (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260925 03:03:51.288666 24606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25102ba08e4e49388f6176badcd10a25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:51.289543 24608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d7691c6b475428c82d6181d693d26f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:51.298231 24664 raft_consensus.cc:348] T 442d4a29b1d743599f4b6b4871e26668 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:51.302228 24664 raft_consensus.cc:374] T 442d4a29b1d743599f4b6b4871e26668 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:51.302567 24664 raft_consensus.cc:729] T 442d4a29b1d743599f4b6b4871e26668 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Initialized, Role: FOLLOWER
I20260925 03:03:51.304667 24612 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:38550 (request call id 42) took 1059 ms. Trace:
I20260925 03:03:51.305125 24612 rpcz_store.cc:276] 0925 03:03:50.245624 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:03:51.010709 (+765085us) service_pool.cc:227] Handling call
0925 03:03:51.027746 (+ 17037us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0925 03:03:51.027847 (+ 101us) ts_tablet_manager.cc:676] Creating new metadata...
0925 03:03:51.215372 (+187525us) tablet_metadata.cc:641] Metadata flushed
0925 03:03:51.304143 (+ 88771us) tablet_replica.cc:205] Creating consensus instance
0925 03:03:51.304605 (+ 462us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:03:51.308996 24664 consensus_queue.cc:260] T 442d4a29b1d743599f4b6b4871e26668 P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:51.312852 24663 tablet_bootstrap.cc:492] T b1e85248d3b04b979dd080caeeb02fdb P a04a37fc4aa7460cbfdbeeb4e451e1f5: Bootstrap starting.
I20260925 03:03:51.313139 24662 tablet_bootstrap.cc:654] T 442d4a29b1d743599f4b6b4871e26668 P eeab93abfdb141a895002ec9f312296c: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:51.314570 24664 ts_tablet_manager.cc:1428] T 442d4a29b1d743599f4b6b4871e26668 P 869b0a8575b54d08ae2817fad5dd42d5: Time spent starting tablet: real 0.033s user 0.003s sys 0.005s
I20260925 03:03:51.288231 24533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6b6ec010caf4689b8c90449764f5cf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:51.320588 24538 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:36784 (request call id 51) took 1008 ms. Trace:
I20260925 03:03:51.321312 24455 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:55552 (request call id 37) took 1069 ms. Trace:
I20260925 03:03:51.321357 24538 rpcz_store.cc:276] 0925 03:03:50.312592 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:03:51.185702 (+873110us) service_pool.cc:227] Handling call
0925 03:03:51.197609 (+ 11907us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0925 03:03:51.197689 (+ 80us) ts_tablet_manager.cc:676] Creating new metadata...
0925 03:03:51.275625 (+ 77936us) tablet_metadata.cc:641] Metadata flushed
0925 03:03:51.320020 (+ 44395us) tablet_replica.cc:205] Creating consensus instance
0925 03:03:51.320523 (+ 503us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:03:51.260555 24611 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:38550 (request call id 43) took 1014 ms. Trace:
I20260925 03:03:51.325192 24611 rpcz_store.cc:276] 0925 03:03:50.245960 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:03:51.012257 (+766297us) service_pool.cc:227] Handling call
0925 03:03:51.021510 (+ 9253us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0925 03:03:51.021630 (+ 120us) ts_tablet_manager.cc:676] Creating new metadata...
0925 03:03:51.135918 (+114288us) tablet_metadata.cc:641] Metadata flushed
0925 03:03:51.260024 (+124106us) tablet_replica.cc:205] Creating consensus instance
0925 03:03:51.260488 (+ 464us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:03:51.325642 24458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac9357d144aa49789faaf31fe5e6bf89. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:51.327914 24611 tablet_service.cc:1467] Processing CreateTablet for tablet ceab6b0fccd54e909a64f66fe81a9a1f (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260925 03:03:51.328662 24463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9c33fd100ff44c1a3247319a324a044. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:51.329290 24611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ceab6b0fccd54e909a64f66fe81a9a1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:51.332513 24612 tablet_service.cc:1467] Processing CreateTablet for tablet 6c81a1b3b616492ab6c964f45af36077 (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260925 03:03:51.333881 24612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c81a1b3b616492ab6c964f45af36077. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:51.301041 24605 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:38550 (request call id 41) took 1055 ms. Trace:
I20260925 03:03:51.338538 24610 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:38550 (request call id 48) took 1040 ms. Trace:
I20260925 03:03:51.338953 24610 rpcz_store.cc:276] 0925 03:03:50.297970 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:03:51.228267 (+930297us) service_pool.cc:227] Handling call
0925 03:03:51.230155 (+ 1888us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0925 03:03:51.230381 (+ 226us) ts_tablet_manager.cc:676] Creating new metadata...
0925 03:03:51.302782 (+ 72401us) tablet_metadata.cc:641] Metadata flushed
0925 03:03:51.338022 (+ 35240us) tablet_replica.cc:205] Creating consensus instance
0925 03:03:51.338478 (+ 456us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:03:51.332590 24459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11bb46bce3564841b7674e6a693bd2e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:51.331230 24538 tablet_service.cc:1467] Processing CreateTablet for tablet cc2df40db08649c9a75e90000b878559 (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260925 03:03:51.342968 24538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc2df40db08649c9a75e90000b878559. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:51.346100 24663 tablet_bootstrap.cc:654] T b1e85248d3b04b979dd080caeeb02fdb P a04a37fc4aa7460cbfdbeeb4e451e1f5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:51.347488 24606 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:38550 (request call id 49) took 1049 ms. Trace:
I20260925 03:03:51.347929 24606 rpcz_store.cc:276] 0925 03:03:50.298271 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:03:51.257271 (+959000us) service_pool.cc:227] Handling call
0925 03:03:51.284626 (+ 27355us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0925 03:03:51.288326 (+ 3700us) ts_tablet_manager.cc:676] Creating new metadata...
0925 03:03:51.299244 (+ 10918us) tablet_metadata.cc:641] Metadata flushed
0925 03:03:51.346941 (+ 47697us) tablet_replica.cc:205] Creating consensus instance
0925 03:03:51.347430 (+ 489us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:03:51.349687 24610 tablet_service.cc:1467] Processing CreateTablet for tablet e6b6ec010caf4689b8c90449764f5cf9 (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260925 03:03:51.350929 24610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6b6ec010caf4689b8c90449764f5cf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:51.352295 24607 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:38550 (request call id 47) took 1054 ms. Trace:
W20260925 03:03:51.353884 24418 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260925 03:03:51.364938 24607 rpcz_store.cc:276] 0925 03:03:50.297596 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:03:51.219119 (+921523us) service_pool.cc:227] Handling call
0925 03:03:51.221581 (+ 2462us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0925 03:03:51.221716 (+ 135us) ts_tablet_manager.cc:676] Creating new metadata...
0925 03:03:51.297415 (+ 75699us) tablet_metadata.cc:641] Metadata flushed
0925 03:03:51.351781 (+ 54366us) tablet_replica.cc:205] Creating consensus instance
0925 03:03:51.352238 (+ 457us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:03:51.354562 24604 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:38550 (request call id 44) took 1108 ms. Trace:
I20260925 03:03:51.321810 24455 rpcz_store.cc:276] 0925 03:03:50.251909 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:03:50.867402 (+615493us) service_pool.cc:227] Handling call
0925 03:03:50.931624 (+ 64222us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0925 03:03:50.938333 (+ 6709us) ts_tablet_manager.cc:676] Creating new metadata...
0925 03:03:51.166836 (+228503us) tablet_metadata.cc:641] Metadata flushed
0925 03:03:51.222405 (+ 55569us) tablet_replica.cc:205] Creating consensus instance
0925 03:03:51.321251 (+ 98846us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:03:51.336275 24605 rpcz_store.cc:276] 0925 03:03:50.245218 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:03:50.996784 (+751566us) service_pool.cc:227] Handling call
0925 03:03:50.998534 (+ 1750us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0925 03:03:50.998631 (+ 97us) ts_tablet_manager.cc:676] Creating new metadata...
0925 03:03:51.127839 (+129208us) tablet_metadata.cc:641] Metadata flushed
0925 03:03:51.300513 (+172674us) tablet_replica.cc:205] Creating consensus instance
0925 03:03:51.300982 (+ 469us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:03:51.374830 24605 tablet_service.cc:1467] Processing CreateTablet for tablet 35992316cb8d4c028333388eff02ecde (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260925 03:03:51.376099 24605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35992316cb8d4c028333388eff02ecde. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:51.362874 24603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:38550 (request call id 45) took 1116 ms. Trace:
I20260925 03:03:51.380836 24603 rpcz_store.cc:276] 0925 03:03:50.246552 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:03:51.172981 (+926429us) service_pool.cc:227] Handling call
0925 03:03:51.226067 (+ 53086us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0925 03:03:51.226157 (+ 90us) ts_tablet_manager.cc:676] Creating new metadata...
0925 03:03:51.340872 (+114715us) tablet_metadata.cc:641] Metadata flushed
0925 03:03:51.362353 (+ 21481us) tablet_replica.cc:205] Creating consensus instance
0925 03:03:51.362803 (+ 450us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:03:51.381428 24664 tablet_bootstrap.cc:492] T f22ddd7d1c444dc58f45a1fb0b3e02bc P 869b0a8575b54d08ae2817fad5dd42d5: Bootstrap starting.
I20260925 03:03:51.386231 24662 tablet_bootstrap.cc:492] T 442d4a29b1d743599f4b6b4871e26668 P eeab93abfdb141a895002ec9f312296c: No bootstrap required, opened a new log
I20260925 03:03:51.386755 24662 ts_tablet_manager.cc:1397] T 442d4a29b1d743599f4b6b4871e26668 P eeab93abfdb141a895002ec9f312296c: Time spent bootstrapping tablet: real 0.109s user 0.012s sys 0.001s
I20260925 03:03:51.387166 24538 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:36784 (request call id 54) took 1027 ms. Trace:
I20260925 03:03:51.387699 24454 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:55552 (request call id 41) took 1090 ms. Trace:
I20260925 03:03:51.387837 24462 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:55552 (request call id 43) took 1088 ms. Trace:
I20260925 03:03:51.388125 24454 rpcz_store.cc:276] 0925 03:03:50.297218 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:03:51.220969 (+923751us) service_pool.cc:227] Handling call
0925 03:03:51.222667 (+ 1698us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0925 03:03:51.222751 (+ 84us) ts_tablet_manager.cc:676] Creating new metadata...
0925 03:03:51.317092 (+ 94341us) tablet_metadata.cc:641] Metadata flushed
0925 03:03:51.349887 (+ 32795us) tablet_replica.cc:205] Creating consensus instance
0925 03:03:51.387641 (+ 37754us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5376}
I20260925 03:03:51.388171 24462 rpcz_store.cc:276] 0925 03:03:50.299680 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:03:51.018087 (+718407us) service_pool.cc:227] Handling call
0925 03:03:51.019875 (+ 1788us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0925 03:03:51.020005 (+ 130us) ts_tablet_manager.cc:676] Creating new metadata...
0925 03:03:51.232711 (+212706us) tablet_metadata.cc:641] Metadata flushed
0925 03:03:51.364561 (+131850us) tablet_replica.cc:205] Creating consensus instance
0925 03:03:51.387783 (+ 23222us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:03:51.387543 24538 rpcz_store.cc:276] 0925 03:03:50.360045 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:03:51.330400 (+970355us) service_pool.cc:227] Handling call
0925 03:03:51.342521 (+ 12121us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0925 03:03:51.342616 (+ 95us) ts_tablet_manager.cc:676] Creating new metadata...
0925 03:03:51.374600 (+ 31984us) tablet_metadata.cc:641] Metadata flushed
0925 03:03:51.386647 (+ 12047us) tablet_replica.cc:205] Creating consensus instance
0925 03:03:51.387100 (+ 453us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:03:51.389621 24662 raft_consensus.cc:348] T 442d4a29b1d743599f4b6b4871e26668 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:51.390368 24662 raft_consensus.cc:374] T 442d4a29b1d743599f4b6b4871e26668 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:51.390278 24454 tablet_service.cc:1467] Processing CreateTablet for tablet b5f9b503b0ab43999d582015a367441e (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260925 03:03:51.390777 24662 raft_consensus.cc:729] T 442d4a29b1d743599f4b6b4871e26668 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Initialized, Role: FOLLOWER
I20260925 03:03:51.390475 24462 tablet_service.cc:1467] Processing CreateTablet for tablet 25102ba08e4e49388f6176badcd10a25 (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260925 03:03:51.392321 24530 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:36784 (request call id 42) took 1123 ms. Trace:
I20260925 03:03:51.394179 24536 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:36784 (request call id 43) took 1125 ms. Trace:
I20260925 03:03:51.392863 24538 tablet_service.cc:1467] Processing CreateTablet for tablet 35992316cb8d4c028333388eff02ecde (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260925 03:03:51.396104 24529 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:36784 (request call id 52) took 1083 ms. Trace:
I20260925 03:03:51.376147 24456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e5819c15eac4544a57e5ee6c9d4f117. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:51.418483 24607 tablet_service.cc:1467] Processing CreateTablet for tablet cc2df40db08649c9a75e90000b878559 (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260925 03:03:51.401777 24461 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:55552 (request call id 44) took 1101 ms. Trace:
I20260925 03:03:51.372891 24604 rpcz_store.cc:276] 0925 03:03:50.246259 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:03:51.166854 (+920595us) service_pool.cc:227] Handling call
0925 03:03:51.168650 (+ 1796us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0925 03:03:51.168795 (+ 145us) ts_tablet_manager.cc:676] Creating new metadata...
0925 03:03:51.295024 (+126229us) tablet_metadata.cc:641] Metadata flushed
0925 03:03:51.353951 (+ 58927us) tablet_replica.cc:205] Creating consensus instance
0925 03:03:51.354500 (+ 549us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:03:51.421023 24538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35992316cb8d4c028333388eff02ecde. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:51.420954 24461 rpcz_store.cc:276] 0925 03:03:50.299973 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:03:50.920136 (+620163us) service_pool.cc:227] Handling call
0925 03:03:51.056268 (+136132us) spinlock_profiling.cc:243] Waited 8.05 ms on lock 0x7b5800001598. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f43733ecf11 00007f4373570851 00007f4373570702 00007f437c460b09 00007f437b39c730 00007f437b47ed21 00007f437b45c1e3 00007f437b40d869 00007f4374a0fc44 00007f4374a0fbd2 00007f4374a0fb01
0925 03:03:51.056502 (+ 234us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0925 03:03:51.056622 (+ 120us) ts_tablet_manager.cc:676] Creating new metadata...
0925 03:03:51.228483 (+171861us) tablet_metadata.cc:641] Metadata flushed
0925 03:03:51.369381 (+140898us) tablet_replica.cc:205] Creating consensus instance
0925 03:03:51.401250 (+ 31869us) spinlock_profiling.cc:243] Waited 904 us on lock 0x7b5800001598. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f43733ecf11 00007f4373570851 00007f4373570e82 00007f437c4898b9 00007f437b39c5d8 00007f437b460063 00007f437b45be2f 00007f437b45c5ed 00007f437b40d869 00007f4374a0fc44 00007f4374a0fbd2
0925 03:03:51.401706 (+ 456us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":20595840}
I20260925 03:03:51.422093 24604 tablet_service.cc:1467] Processing CreateTablet for tablet 100c113a9fd84cc8adf2ec7b33ea0093 (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260925 03:03:51.422530 24607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc2df40db08649c9a75e90000b878559. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:51.423246 24604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 100c113a9fd84cc8adf2ec7b33ea0093. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:51.423684 24461 tablet_service.cc:1467] Processing CreateTablet for tablet ceab6b0fccd54e909a64f66fe81a9a1f (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260925 03:03:51.403430 24532 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:36784 (request call id 44) took 1094 ms. Trace:
I20260925 03:03:51.424537 24532 rpcz_store.cc:276] 0925 03:03:50.308952 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:03:51.036322 (+727370us) service_pool.cc:227] Handling call
0925 03:03:51.093187 (+ 56865us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0925 03:03:51.093300 (+ 113us) ts_tablet_manager.cc:676] Creating new metadata...
0925 03:03:51.316174 (+222874us) tablet_metadata.cc:641] Metadata flushed
0925 03:03:51.402932 (+ 86758us) tablet_replica.cc:205] Creating consensus instance
0925 03:03:51.403375 (+ 443us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:03:51.407403 24535 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:36784 (request call id 48) took 1095 ms. Trace:
I20260925 03:03:51.426273 24535 rpcz_store.cc:276] 0925 03:03:50.311649 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:03:51.151554 (+839905us) service_pool.cc:227] Handling call
0925 03:03:51.197366 (+ 45812us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0925 03:03:51.197478 (+ 112us) ts_tablet_manager.cc:676] Creating new metadata...
0925 03:03:51.332657 (+135179us) tablet_metadata.cc:641] Metadata flushed
0925 03:03:51.406863 (+ 74206us) tablet_replica.cc:205] Creating consensus instance
0925 03:03:51.407339 (+ 476us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:03:51.411561 24608 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:38550 (request call id 50) took 1113 ms. Trace:
I20260925 03:03:51.427958 24608 rpcz_store.cc:276] 0925 03:03:50.298649 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:03:51.287273 (+988624us) service_pool.cc:227] Handling call
0925 03:03:51.289102 (+ 1829us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0925 03:03:51.289192 (+ 90us) ts_tablet_manager.cc:676] Creating new metadata...
0925 03:03:51.320303 (+ 31111us) tablet_metadata.cc:641] Metadata flushed
0925 03:03:51.411038 (+ 90735us) tablet_replica.cc:205] Creating consensus instance
0925 03:03:51.411503 (+ 465us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:03:51.413405 24537 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:36784 (request call id 49) took 1101 ms. Trace:
I20260925 03:03:51.429733 24537 rpcz_store.cc:276] 0925 03:03:50.312011 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:03:51.159505 (+847494us) service_pool.cc:227] Handling call
0925 03:03:51.201564 (+ 42059us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0925 03:03:51.201667 (+ 103us) ts_tablet_manager.cc:676] Creating new metadata...
0925 03:03:51.311367 (+109700us) tablet_metadata.cc:641] Metadata flushed
0925 03:03:51.412922 (+101555us) tablet_replica.cc:205] Creating consensus instance
0925 03:03:51.413348 (+ 426us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:03:51.432308 24537 tablet_service.cc:1467] Processing CreateTablet for tablet 420163a6cdce412ebc21c25fe3835638 (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260925 03:03:51.433621 24537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 420163a6cdce412ebc21c25fe3835638. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:51.414125 24611 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:38550 (request call id 51) took 1115 ms. Trace:
I20260925 03:03:51.413846 24603 tablet_service.cc:1467] Processing CreateTablet for tablet 7b8e0d260a884ecdb20f79de46c36b19 (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260925 03:03:51.437071 24662 consensus_queue.cc:260] T 442d4a29b1d743599f4b6b4871e26668 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:51.419301 24530 rpcz_store.cc:276] 0925 03:03:50.268461 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:03:51.006590 (+738129us) service_pool.cc:227] Handling call
0925 03:03:51.089409 (+ 82819us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0925 03:03:51.089523 (+ 114us) ts_tablet_manager.cc:676] Creating new metadata...
0925 03:03:51.310758 (+221235us) tablet_metadata.cc:641] Metadata flushed
0925 03:03:51.391477 (+ 80719us) tablet_replica.cc:205] Creating consensus instance
0925 03:03:51.392235 (+ 758us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:03:51.401679 24606 tablet_service.cc:1467] Processing CreateTablet for tablet 420163a6cdce412ebc21c25fe3835638 (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260925 03:03:51.438772 24607 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:38550 (request call id 54) took 1087 ms. Trace:
I20260925 03:03:51.415557 24455 tablet_service.cc:1467] Processing CreateTablet for tablet 1d7691c6b475428c82d6181d693d26f3 (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260925 03:03:51.417369 24534 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:36784 (request call id 47) took 1107 ms. Trace:
I20260925 03:03:51.439934 24534 rpcz_store.cc:276] 0925 03:03:50.311218 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:03:51.142236 (+831018us) service_pool.cc:227] Handling call
0925 03:03:51.149817 (+ 7581us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0925 03:03:51.149912 (+ 95us) ts_tablet_manager.cc:676] Creating new metadata...
0925 03:03:51.346538 (+196626us) tablet_metadata.cc:641] Metadata flushed
0925 03:03:51.416807 (+ 70269us) tablet_replica.cc:205] Creating consensus instance
0925 03:03:51.417299 (+ 492us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10236}
I20260925 03:03:51.441027 24535 tablet_service.cc:1467] Processing CreateTablet for tablet 7b8e0d260a884ecdb20f79de46c36b19 (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260925 03:03:51.441804 24534 tablet_service.cc:1467] Processing CreateTablet for tablet 100c113a9fd84cc8adf2ec7b33ea0093 (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260925 03:03:51.424623 24536 rpcz_store.cc:276] 0925 03:03:50.303963 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:03:51.008222 (+704259us) service_pool.cc:227] Handling call
0925 03:03:51.091430 (+ 83208us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0925 03:03:51.091555 (+ 125us) ts_tablet_manager.cc:676] Creating new metadata...
0925 03:03:51.295921 (+204366us) tablet_metadata.cc:641] Metadata flushed
0925 03:03:51.392451 (+ 96530us) tablet_replica.cc:205] Creating consensus instance
0925 03:03:51.394124 (+ 1673us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:03:51.419379 24463 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:55552 (request call id 47) took 1118 ms. Trace:
I20260925 03:03:51.417964 24531 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:36784 (request call id 50) took 1105 ms. Trace:
I20260925 03:03:51.444288 24463 rpcz_store.cc:276] 0925 03:03:50.301791 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:03:51.279591 (+977800us) service_pool.cc:227] Handling call
0925 03:03:51.325865 (+ 46274us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0925 03:03:51.325956 (+ 91us) ts_tablet_manager.cc:676] Creating new metadata...
0925 03:03:51.350099 (+ 24143us) tablet_metadata.cc:641] Metadata flushed
0925 03:03:51.412218 (+ 62119us) tablet_replica.cc:205] Creating consensus instance
0925 03:03:51.418777 (+ 6559us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":101888}
I20260925 03:03:51.419612 24529 rpcz_store.cc:276] 0925 03:03:50.312936 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:03:51.145593 (+832657us) service_pool.cc:227] Handling call
0925 03:03:51.147419 (+ 1826us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0925 03:03:51.147516 (+ 97us) ts_tablet_manager.cc:676] Creating new metadata...
0925 03:03:51.302839 (+155323us) tablet_metadata.cc:641] Metadata flushed
0925 03:03:51.395449 (+ 92610us) tablet_replica.cc:205] Creating consensus instance
0925 03:03:51.396049 (+ 600us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:03:51.444869 24531 rpcz_store.cc:276] 0925 03:03:50.312294 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:03:51.161142 (+848848us) service_pool.cc:227] Handling call
0925 03:03:51.162768 (+ 1626us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0925 03:03:51.162963 (+ 195us) ts_tablet_manager.cc:676] Creating new metadata...
0925 03:03:51.289452 (+126489us) tablet_metadata.cc:641] Metadata flushed
0925 03:03:51.417511 (+128059us) tablet_replica.cc:205] Creating consensus instance
0925 03:03:51.417910 (+ 399us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:03:51.446969 24463 tablet_service.cc:1467] Processing CreateTablet for tablet 6c81a1b3b616492ab6c964f45af36077 (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260925 03:03:51.398105 24457 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:55552 (request call id 38) took 1144 ms. Trace:
I20260925 03:03:51.420557 24454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5f9b503b0ab43999d582015a367441e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:51.438992 24603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b8e0d260a884ecdb20f79de46c36b19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:51.439481 24611 rpcz_store.cc:276] 0925 03:03:50.298997 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:03:51.326924 (+1027927us) service_pool.cc:227] Handling call
0925 03:03:51.328836 (+ 1912us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0925 03:03:51.328933 (+ 97us) ts_tablet_manager.cc:676] Creating new metadata...
0925 03:03:51.407589 (+ 78656us) tablet_metadata.cc:641] Metadata flushed
0925 03:03:51.413658 (+ 6069us) tablet_replica.cc:205] Creating consensus instance
0925 03:03:51.414075 (+ 417us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:03:51.449043 24460 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:55552 (request call id 40) took 1152 ms. Trace:
I20260925 03:03:51.449150 24535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b8e0d260a884ecdb20f79de46c36b19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:51.451213 24534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 100c113a9fd84cc8adf2ec7b33ea0093. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:51.449448 24460 rpcz_store.cc:276] 0925 03:03:50.296866 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:03:51.139344 (+842478us) service_pool.cc:227] Handling call
0925 03:03:51.211930 (+ 72586us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0925 03:03:51.212057 (+ 127us) ts_tablet_manager.cc:676] Creating new metadata...
0925 03:03:51.377835 (+165778us) tablet_metadata.cc:641] Metadata flushed
0925 03:03:51.386403 (+ 8568us) tablet_replica.cc:205] Creating consensus instance
0925 03:03:51.419969 (+ 33566us) spinlock_profiling.cc:243] Waited 6.47 ms on lock 0x7b5800001598. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f43733ecf11 00007f4373570851 00007f4373570e82 00007f437c4898b9 00007f437b39c5d8 00007f437b460063 00007f437b45be2f 00007f437b45c5ed 00007f437b40d869 00007f4374a0fc44 00007f4374a0fbd2
0925 03:03:51.435634 (+ 15665us) spinlock_profiling.cc:243] Waited 15.1 ms on lock 0x7b5800001598. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f43733ecf11 00007f4373570f98 00007f4373570ebb 00007f437c4898b9 00007f437b39c5d8 00007f437b460063 00007f437b45be2f 00007f437b45c5ed 00007f437b40d869 00007f4374a0fc44 00007f4374a0fbd2
0925 03:03:51.448972 (+ 13338us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":12872,"spinlock_wait_cycles":49695872}
I20260925 03:03:51.449690 24462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25102ba08e4e49388f6176badcd10a25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:51.453804 24460 tablet_service.cc:1467] Processing CreateTablet for tablet e6b6ec010caf4689b8c90449764f5cf9 (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260925 03:03:51.455116 24460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6b6ec010caf4689b8c90449764f5cf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:51.439251 24607 rpcz_store.cc:276] 0925 03:03:50.351320 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:03:51.370263 (+1018943us) service_pool.cc:227] Handling call
0925 03:03:51.422104 (+ 51841us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0925 03:03:51.422192 (+ 88us) ts_tablet_manager.cc:676] Creating new metadata...
0925 03:03:51.431219 (+ 9027us) tablet_metadata.cc:641] Metadata flushed
0925 03:03:51.437921 (+ 6702us) tablet_replica.cc:205] Creating consensus instance
0925 03:03:51.438588 (+ 667us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:03:51.457828 24606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 420163a6cdce412ebc21c25fe3835638. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:51.460382 24533 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:36784 (request call id 53) took 1100 ms. Trace:
I20260925 03:03:51.448167 24457 rpcz_store.cc:276] 0925 03:03:50.253397 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:03:50.881830 (+628433us) service_pool.cc:227] Handling call
0925 03:03:50.913047 (+ 31217us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0925 03:03:50.913141 (+ 94us) ts_tablet_manager.cc:676] Creating new metadata...
0925 03:03:51.195818 (+282677us) tablet_metadata.cc:641] Metadata flushed
0925 03:03:51.397624 (+201806us) tablet_replica.cc:205] Creating consensus instance
0925 03:03:51.398048 (+ 424us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:03:51.460350 24663 tablet_bootstrap.cc:492] T b1e85248d3b04b979dd080caeeb02fdb P a04a37fc4aa7460cbfdbeeb4e451e1f5: No bootstrap required, opened a new log
I20260925 03:03:51.461936 24459 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:55552 (request call id 45) took 1161 ms. Trace:
I20260925 03:03:51.462329 24459 rpcz_store.cc:276] 0925 03:03:50.300264 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:03:51.275388 (+975124us) service_pool.cc:227] Handling call
0925 03:03:51.325744 (+ 50356us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0925 03:03:51.326122 (+ 378us) ts_tablet_manager.cc:676] Creating new metadata...
0925 03:03:51.360164 (+ 34042us) tablet_metadata.cc:641] Metadata flushed
0925 03:03:51.461439 (+101275us) tablet_replica.cc:205] Creating consensus instance
0925 03:03:51.461877 (+ 438us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":55552}
I20260925 03:03:51.436579 24609 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:38550 (request call id 46) took 1189 ms. Trace:
I20260925 03:03:51.465427 24663 ts_tablet_manager.cc:1397] T b1e85248d3b04b979dd080caeeb02fdb P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent bootstrapping tablet: real 0.153s user 0.018s sys 0.028s
I20260925 03:03:51.464174 24457 tablet_service.cc:1467] Processing CreateTablet for tablet cc2df40db08649c9a75e90000b878559 (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260925 03:03:51.465049 24461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ceab6b0fccd54e909a64f66fe81a9a1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:51.467545 24457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc2df40db08649c9a75e90000b878559. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:51.470894 24663 raft_consensus.cc:348] T b1e85248d3b04b979dd080caeeb02fdb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:51.471697 24663 raft_consensus.cc:374] T b1e85248d3b04b979dd080caeeb02fdb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:51.472023 24663 raft_consensus.cc:729] T b1e85248d3b04b979dd080caeeb02fdb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Initialized, Role: FOLLOWER
I20260925 03:03:51.472836 24458 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:55552 (request call id 46) took 1172 ms. Trace:
I20260925 03:03:51.473215 24458 rpcz_store.cc:276] 0925 03:03:50.300536 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:03:51.253615 (+953079us) service_pool.cc:227] Handling call
0925 03:03:51.325107 (+ 71492us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0925 03:03:51.325244 (+ 137us) ts_tablet_manager.cc:676] Creating new metadata...
0925 03:03:51.339882 (+ 14638us) tablet_metadata.cc:641] Metadata flushed
0925 03:03:51.418205 (+ 78323us) tablet_replica.cc:205] Creating consensus instance
0925 03:03:51.472758 (+ 54553us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":95616}
I20260925 03:03:51.473760 24533 rpcz_store.cc:276] 0925 03:03:50.359713 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:03:51.285999 (+926286us) service_pool.cc:227] Handling call
0925 03:03:51.287707 (+ 1708us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0925 03:03:51.287834 (+ 127us) ts_tablet_manager.cc:676] Creating new metadata...
0925 03:03:51.397206 (+109372us) tablet_metadata.cc:641] Metadata flushed
0925 03:03:51.451353 (+ 54147us) tablet_replica.cc:205] Creating consensus instance
0925 03:03:51.460312 (+ 8959us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:03:51.475167 24458 tablet_service.cc:1467] Processing CreateTablet for tablet 420163a6cdce412ebc21c25fe3835638 (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260925 03:03:51.476410 24458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 420163a6cdce412ebc21c25fe3835638. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:51.472854 24663 consensus_queue.cc:260] T b1e85248d3b04b979dd080caeeb02fdb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:51.478598 24664 tablet_bootstrap.cc:654] T f22ddd7d1c444dc58f45a1fb0b3e02bc P 869b0a8575b54d08ae2817fad5dd42d5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:51.483877 24663 ts_tablet_manager.cc:1428] T b1e85248d3b04b979dd080caeeb02fdb P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent starting tablet: real 0.018s user 0.006s sys 0.001s
I20260925 03:03:51.493369 24663 tablet_bootstrap.cc:492] T 7993f9c3e2a44f408bbf5d1e395e2306 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Bootstrap starting.
I20260925 03:03:51.497198 24455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d7691c6b475428c82d6181d693d26f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:51.485657 24610 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:38550 (request call id 53) took 1158 ms. Trace:
I20260925 03:03:51.489380 24662 ts_tablet_manager.cc:1428] T 442d4a29b1d743599f4b6b4871e26668 P eeab93abfdb141a895002ec9f312296c: Time spent starting tablet: real 0.102s user 0.006s sys 0.000s
I20260925 03:03:51.497778 24610 rpcz_store.cc:276] 0925 03:03:50.327623 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:03:51.347480 (+1019857us) service_pool.cc:227] Handling call
0925 03:03:51.350467 (+ 2987us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0925 03:03:51.350557 (+ 90us) ts_tablet_manager.cc:676] Creating new metadata...
0925 03:03:51.429741 (+ 79184us) tablet_metadata.cc:641] Metadata flushed
0925 03:03:51.485156 (+ 55415us) tablet_replica.cc:205] Creating consensus instance
0925 03:03:51.485597 (+ 441us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:03:51.501644 24663 tablet_bootstrap.cc:654] T 7993f9c3e2a44f408bbf5d1e395e2306 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:51.510192 24463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c81a1b3b616492ab6c964f45af36077. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:51.489645 24612 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:38550 (request call id 52) took 1175 ms. Trace:
I20260925 03:03:51.505612 24662 tablet_bootstrap.cc:492] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c: Bootstrap starting.
I20260925 03:03:51.515323 24605 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:38550 (request call id 55) took 1089 ms. Trace:
I20260925 03:03:51.465008 24609 rpcz_store.cc:276] 0925 03:03:50.246882 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:03:51.204011 (+957129us) service_pool.cc:227] Handling call
0925 03:03:51.222991 (+ 18980us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0925 03:03:51.223098 (+ 107us) ts_tablet_manager.cc:676] Creating new metadata...
0925 03:03:51.339017 (+115919us) tablet_metadata.cc:641] Metadata flushed
0925 03:03:51.436073 (+ 97056us) tablet_replica.cc:205] Creating consensus instance
0925 03:03:51.436518 (+ 445us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:03:51.521870 24456 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:55552 (request call id 42) took 1222 ms. Trace:
I20260925 03:03:51.522361 24456 rpcz_store.cc:276] 0925 03:03:50.299351 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:03:51.249746 (+950395us) service_pool.cc:227] Handling call
0925 03:03:51.324840 (+ 75094us) spinlock_profiling.cc:243] Waited 2.71 ms on lock 0x7b5800001598. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f43733ecf11 00007f4373570851 00007f4373570bce 000000000034aea9 00007f437b39ca30 00007f437b47ed3c 00007f437b45c1e3 00007f437b40d869 00007f4374a0fc44 00007f4374a0fbd2 00007f4374a0fb01
0925 03:03:51.375624 (+ 50784us) spinlock_profiling.cc:243] Waited 2.22 ms on lock 0x7b5800001598. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f43733ecf11 00007f4373570f98 00007f4373570ebb 00007f437c4898b9 00007f437b39c5d8 00007f437b45c2a0 00007f437b40d869 00007f4374a0fc44 00007f4374a0fbd2 00007f4374a0fb01 00007f4374a0fa7c
0925 03:03:51.375669 (+ 45us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0925 03:03:51.375793 (+ 124us) ts_tablet_manager.cc:676] Creating new metadata...
0925 03:03:51.480488 (+104695us) tablet_metadata.cc:641] Metadata flushed
0925 03:03:51.521142 (+ 40654us) tablet_replica.cc:205] Creating consensus instance
0925 03:03:51.521811 (+ 669us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":11336832}
I20260925 03:03:51.464177 24459 tablet_service.cc:1467] Processing CreateTablet for tablet 35992316cb8d4c028333388eff02ecde (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260925 03:03:51.516798 24612 rpcz_store.cc:276] 0925 03:03:50.314375 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:03:51.331674 (+1017299us) service_pool.cc:227] Handling call
0925 03:03:51.333450 (+ 1776us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0925 03:03:51.333548 (+ 98us) ts_tablet_manager.cc:676] Creating new metadata...
0925 03:03:51.441009 (+107461us) tablet_metadata.cc:641] Metadata flushed
0925 03:03:51.489107 (+ 48098us) tablet_replica.cc:205] Creating consensus instance
0925 03:03:51.489584 (+ 477us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:03:51.517499 24538 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:36784 (request call id 55) took 1080 ms. Trace:
I20260925 03:03:51.528936 24538 rpcz_store.cc:276] 0925 03:03:50.436897 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:03:51.391541 (+954644us) service_pool.cc:227] Handling call
0925 03:03:51.420562 (+ 29021us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0925 03:03:51.420656 (+ 94us) ts_tablet_manager.cc:676] Creating new metadata...
0925 03:03:51.466247 (+ 45591us) tablet_metadata.cc:641] Metadata flushed
0925 03:03:51.517023 (+ 50776us) tablet_replica.cc:205] Creating consensus instance
0925 03:03:51.517442 (+ 419us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:03:51.530534 24664 tablet_bootstrap.cc:492] T f22ddd7d1c444dc58f45a1fb0b3e02bc P 869b0a8575b54d08ae2817fad5dd42d5: No bootstrap required, opened a new log
I20260925 03:03:51.531062 24664 ts_tablet_manager.cc:1397] T f22ddd7d1c444dc58f45a1fb0b3e02bc P 869b0a8575b54d08ae2817fad5dd42d5: Time spent bootstrapping tablet: real 0.150s user 0.014s sys 0.001s
I20260925 03:03:51.519424 24662 tablet_bootstrap.cc:654] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:51.531428 24537 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:36784 (request call id 56) took 1094 ms. Trace:
I20260925 03:03:51.531793 24537 rpcz_store.cc:276] 0925 03:03:50.437274 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:03:51.431387 (+994113us) service_pool.cc:227] Handling call
0925 03:03:51.433150 (+ 1763us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0925 03:03:51.433248 (+ 98us) ts_tablet_manager.cc:676] Creating new metadata...
0925 03:03:51.490772 (+ 57524us) tablet_metadata.cc:641] Metadata flushed
0925 03:03:51.530940 (+ 40168us) tablet_replica.cc:205] Creating consensus instance
0925 03:03:51.531373 (+ 433us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:03:51.506985 24603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:38550 (request call id 57) took 1058 ms. Trace:
I20260925 03:03:51.533479 24603 rpcz_store.cc:276] 0925 03:03:50.448648 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:03:51.399848 (+951200us) service_pool.cc:227] Handling call
0925 03:03:51.438067 (+ 38219us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0925 03:03:51.438293 (+ 226us) ts_tablet_manager.cc:676] Creating new metadata...
0925 03:03:51.481461 (+ 43168us) tablet_metadata.cc:641] Metadata flushed
0925 03:03:51.506459 (+ 24998us) tablet_replica.cc:205] Creating consensus instance
0925 03:03:51.506926 (+ 467us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:03:51.523484 24462 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:55552 (request call id 49) took 1221 ms. Trace:
I20260925 03:03:51.515726 24605 rpcz_store.cc:276] 0925 03:03:50.425741 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:03:51.373930 (+948189us) service_pool.cc:227] Handling call
0925 03:03:51.375649 (+ 1719us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0925 03:03:51.375743 (+ 94us) ts_tablet_manager.cc:676] Creating new metadata...
0925 03:03:51.469200 (+ 93457us) tablet_metadata.cc:641] Metadata flushed
0925 03:03:51.514766 (+ 45566us) tablet_replica.cc:205] Creating consensus instance
0925 03:03:51.515262 (+ 496us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:03:51.536198 24462 rpcz_store.cc:276] 0925 03:03:50.302510 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:03:51.389527 (+1087017us) service_pool.cc:227] Handling call
0925 03:03:51.435971 (+ 46444us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0925 03:03:51.436245 (+ 274us) ts_tablet_manager.cc:676] Creating new metadata...
0925 03:03:51.482781 (+ 46536us) tablet_metadata.cc:641] Metadata flushed
0925 03:03:51.522988 (+ 40207us) tablet_replica.cc:205] Creating consensus instance
0925 03:03:51.523428 (+ 440us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:03:51.536990 24459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35992316cb8d4c028333388eff02ecde. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:51.538803 24462 tablet_service.cc:1467] Processing CreateTablet for tablet 7b8e0d260a884ecdb20f79de46c36b19 (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260925 03:03:51.540114 24462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b8e0d260a884ecdb20f79de46c36b19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:51.542920 24460 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:55552 (request call id 53) took 1214 ms. Trace:
I20260925 03:03:51.543291 24460 rpcz_store.cc:276] 0925 03:03:50.328898 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:03:51.452942 (+1124044us) service_pool.cc:227] Handling call
0925 03:03:51.454595 (+ 1653us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0925 03:03:51.454688 (+ 93us) ts_tablet_manager.cc:676] Creating new metadata...
0925 03:03:51.493042 (+ 38354us) tablet_metadata.cc:641] Metadata flushed
0925 03:03:51.542372 (+ 49330us) tablet_replica.cc:205] Creating consensus instance
0925 03:03:51.542863 (+ 491us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:03:51.558682 24664 raft_consensus.cc:348] T f22ddd7d1c444dc58f45a1fb0b3e02bc P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:51.559051 24667 raft_consensus.cc:482] T 442d4a29b1d743599f4b6b4871e26668 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:51.559499 24664 raft_consensus.cc:374] T f22ddd7d1c444dc58f45a1fb0b3e02bc P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:51.559880 24664 raft_consensus.cc:729] T f22ddd7d1c444dc58f45a1fb0b3e02bc P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Initialized, Role: FOLLOWER
I20260925 03:03:51.559672 24667 raft_consensus.cc:504] T 442d4a29b1d743599f4b6b4871e26668 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:51.561955 24667 leader_election.cc:288] T 442d4a29b1d743599f4b6b4871e26668 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:51.562223 24535 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:36784 (request call id 57) took 1052 ms. Trace:
I20260925 03:03:51.562633 24535 rpcz_store.cc:276] 0925 03:03:50.510303 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:03:51.440070 (+929767us) service_pool.cc:227] Handling call
0925 03:03:51.441848 (+ 1778us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0925 03:03:51.441943 (+ 95us) ts_tablet_manager.cc:676] Creating new metadata...
0925 03:03:51.517757 (+ 75814us) tablet_metadata.cc:641] Metadata flushed
0925 03:03:51.561719 (+ 43962us) tablet_replica.cc:205] Creating consensus instance
0925 03:03:51.562166 (+ 447us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:03:51.564041 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442d4a29b1d743599f4b6b4871e26668" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:03:51.564785 24473 raft_consensus.cc:2454] T 442d4a29b1d743599f4b6b4871e26668 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 0.
I20260925 03:03:51.566337 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442d4a29b1d743599f4b6b4871e26668" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:03:51.567013 24622 raft_consensus.cc:2454] T 442d4a29b1d743599f4b6b4871e26668 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 0.
I20260925 03:03:51.570592 24514 leader_election.cc:302] T 442d4a29b1d743599f4b6b4871e26668 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:03:51.568742 24664 consensus_queue.cc:260] T f22ddd7d1c444dc58f45a1fb0b3e02bc P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:51.573622 24667 raft_consensus.cc:2781] T 442d4a29b1d743599f4b6b4871e26668 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 03:03:51.566956 24455 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:55552 (request call id 50) took 1239 ms. Trace:
I20260925 03:03:51.574014 24667 raft_consensus.cc:482] T 442d4a29b1d743599f4b6b4871e26668 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:03:51.576136 24667 raft_consensus.cc:3037] T 442d4a29b1d743599f4b6b4871e26668 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:51.570279 24454 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:55552 (request call id 48) took 1268 ms. Trace:
I20260925 03:03:51.578195 24454 rpcz_store.cc:276] 0925 03:03:50.302189 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:03:51.389366 (+1087177us) service_pool.cc:227] Handling call
0925 03:03:51.399453 (+ 10087us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0925 03:03:51.399545 (+ 92us) ts_tablet_manager.cc:676] Creating new metadata...
0925 03:03:51.490263 (+ 90718us) tablet_metadata.cc:641] Metadata flushed
0925 03:03:51.567804 (+ 77541us) tablet_replica.cc:205] Creating consensus instance
0925 03:03:51.569886 (+ 2082us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:03:51.580622 24534 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:36784 (request call id 58) took 1070 ms. Trace:
I20260925 03:03:51.581032 24534 rpcz_store.cc:276] 0925 03:03:50.510676 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:03:51.440948 (+930272us) service_pool.cc:227] Handling call
0925 03:03:51.442524 (+ 1576us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0925 03:03:51.442621 (+ 97us) ts_tablet_manager.cc:676] Creating new metadata...
0925 03:03:51.519660 (+ 77039us) tablet_metadata.cc:641] Metadata flushed
0925 03:03:51.580118 (+ 60458us) tablet_replica.cc:205] Creating consensus instance
0925 03:03:51.580563 (+ 445us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:03:51.583774 24459 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:55552 (request call id 55) took 1178 ms. Trace:
I20260925 03:03:51.584137 24459 rpcz_store.cc:276] 0925 03:03:50.405600 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:03:51.463331 (+1057731us) service_pool.cc:227] Handling call
0925 03:03:51.536360 (+ 73029us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0925 03:03:51.536486 (+ 126us) ts_tablet_manager.cc:676] Creating new metadata...
0925 03:03:51.576504 (+ 40018us) tablet_metadata.cc:641] Metadata flushed
0925 03:03:51.583248 (+ 6744us) tablet_replica.cc:205] Creating consensus instance
0925 03:03:51.583715 (+ 467us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:03:51.558619 24604 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:38550 (request call id 58) took 1080 ms. Trace:
I20260925 03:03:51.586233 24664 ts_tablet_manager.cc:1428] T f22ddd7d1c444dc58f45a1fb0b3e02bc P 869b0a8575b54d08ae2817fad5dd42d5: Time spent starting tablet: real 0.055s user 0.009s sys 0.000s
I20260925 03:03:51.586247 24604 rpcz_store.cc:276] 0925 03:03:50.478319 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:03:51.421249 (+942930us) service_pool.cc:227] Handling call
0925 03:03:51.422831 (+ 1582us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0925 03:03:51.422921 (+ 90us) ts_tablet_manager.cc:676] Creating new metadata...
0925 03:03:51.447314 (+ 24393us) tablet_metadata.cc:641] Metadata flushed
0925 03:03:51.558074 (+110760us) tablet_replica.cc:205] Creating consensus instance
0925 03:03:51.558557 (+ 483us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:03:51.587270 24664 tablet_bootstrap.cc:492] T 7993f9c3e2a44f408bbf5d1e395e2306 P 869b0a8575b54d08ae2817fad5dd42d5: Bootstrap starting.
I20260925 03:03:51.587873 24662 tablet_bootstrap.cc:492] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c: No bootstrap required, opened a new log
I20260925 03:03:51.588348 24662 ts_tablet_manager.cc:1397] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c: Time spent bootstrapping tablet: real 0.083s user 0.009s sys 0.009s
I20260925 03:03:51.588495 24462 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:55552 (request call id 57) took 1161 ms. Trace:
I20260925 03:03:51.588893 24462 rpcz_store.cc:276] 0925 03:03:50.426847 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:03:51.537951 (+1111104us) service_pool.cc:227] Handling call
0925 03:03:51.539615 (+ 1664us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0925 03:03:51.539729 (+ 114us) ts_tablet_manager.cc:676] Creating new metadata...
0925 03:03:51.575288 (+ 35559us) tablet_metadata.cc:641] Metadata flushed
0925 03:03:51.588086 (+ 12798us) tablet_replica.cc:205] Creating consensus instance
0925 03:03:51.588452 (+ 366us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:03:51.574674 24455 rpcz_store.cc:276] 0925 03:03:50.327921 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:03:51.414642 (+1086721us) service_pool.cc:227] Handling call
0925 03:03:51.442424 (+ 27782us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0925 03:03:51.444830 (+ 2406us) ts_tablet_manager.cc:676] Creating new metadata...
0925 03:03:51.536498 (+ 91668us) tablet_metadata.cc:641] Metadata flushed
0925 03:03:51.566484 (+ 29986us) tablet_replica.cc:205] Creating consensus instance
0925 03:03:51.566900 (+ 416us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:03:51.592226 24663 tablet_bootstrap.cc:492] T 7993f9c3e2a44f408bbf5d1e395e2306 P a04a37fc4aa7460cbfdbeeb4e451e1f5: No bootstrap required, opened a new log
I20260925 03:03:51.580974 24456 tablet_service.cc:1467] Processing CreateTablet for tablet 100c113a9fd84cc8adf2ec7b33ea0093 (DEFAULT_TABLE table=test-workload [id=f20271173d58474897a63e58ef5a856d]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260925 03:03:51.600577 24461 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:55552 (request call id 51) took 1272 ms. Trace:
I20260925 03:03:51.593032 24664 tablet_bootstrap.cc:654] T 7993f9c3e2a44f408bbf5d1e395e2306 P 869b0a8575b54d08ae2817fad5dd42d5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:51.593389 24667 raft_consensus.cc:504] T 442d4a29b1d743599f4b6b4871e26668 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:51.594794 24457 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:55552 (request call id 54) took 1189 ms. Trace:
I20260925 03:03:51.602326 24457 rpcz_store.cc:276] 0925 03:03:50.405241 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:03:51.462522 (+1057281us) service_pool.cc:227] Handling call
0925 03:03:51.467099 (+ 4577us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0925 03:03:51.467191 (+ 92us) ts_tablet_manager.cc:676] Creating new metadata...
0925 03:03:51.507763 (+ 40572us) tablet_metadata.cc:641] Metadata flushed
0925 03:03:51.594299 (+ 86536us) tablet_replica.cc:205] Creating consensus instance
0925 03:03:51.594738 (+ 439us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:03:51.603480 24667 leader_election.cc:288] T 442d4a29b1d743599f4b6b4871e26668 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:51.604323 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442d4a29b1d743599f4b6b4871e26668" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:03:51.604933 24473 raft_consensus.cc:3037] T 442d4a29b1d743599f4b6b4871e26668 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:51.605165 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442d4a29b1d743599f4b6b4871e26668" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:03:51.605849 24622 raft_consensus.cc:3037] T 442d4a29b1d743599f4b6b4871e26668 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:51.609586 24463 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:55552 (request call id 52) took 1281 ms. Trace:
I20260925 03:03:51.609963 24463 rpcz_store.cc:276] 0925 03:03:50.328546 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:03:51.446080 (+1117534us) service_pool.cc:227] Handling call
0925 03:03:51.447732 (+ 1652us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0925 03:03:51.447824 (+ 92us) ts_tablet_manager.cc:676] Creating new metadata...
0925 03:03:51.601563 (+153739us) tablet_metadata.cc:641] Metadata flushed
0925 03:03:51.609069 (+ 7506us) tablet_replica.cc:205] Creating consensus instance
0925 03:03:51.609526 (+ 457us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:03:51.598318 24606 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:38550 (request call id 56) took 1172 ms. Trace:
I20260925 03:03:51.599054 24662 raft_consensus.cc:348] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:51.611701 24473 raft_consensus.cc:2454] T 442d4a29b1d743599f4b6b4871e26668 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 1.
I20260925 03:03:51.612035 24662 raft_consensus.cc:374] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:51.612469 24662 raft_consensus.cc:729] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Initialized, Role: FOLLOWER
I20260925 03:03:51.612933 24514 leader_election.cc:302] T 442d4a29b1d743599f4b6b4871e26668 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:03:51.613674 24664 tablet_bootstrap.cc:492] T 7993f9c3e2a44f408bbf5d1e395e2306 P 869b0a8575b54d08ae2817fad5dd42d5: No bootstrap required, opened a new log
I20260925 03:03:51.613219 24662 consensus_queue.cc:260] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:51.613801 24667 raft_consensus.cc:2781] T 442d4a29b1d743599f4b6b4871e26668 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:03:51.614152 24664 ts_tablet_manager.cc:1397] T 7993f9c3e2a44f408bbf5d1e395e2306 P 869b0a8575b54d08ae2817fad5dd42d5: Time spent bootstrapping tablet: real 0.027s user 0.015s sys 0.000s
I20260925 03:03:51.611886 24606 rpcz_store.cc:276] 0925 03:03:50.426102 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:03:51.400848 (+974746us) service_pool.cc:227] Handling call
0925 03:03:51.439133 (+ 38285us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0925 03:03:51.439221 (+ 88us) ts_tablet_manager.cc:676] Creating new metadata...
0925 03:03:51.533440 (+ 94219us) tablet_metadata.cc:641] Metadata flushed
0925 03:03:51.597849 (+ 64409us) tablet_replica.cc:205] Creating consensus instance
0925 03:03:51.598260 (+ 411us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:03:51.615809 24622 raft_consensus.cc:2454] T 442d4a29b1d743599f4b6b4871e26668 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 1.
I20260925 03:03:51.617066 24664 raft_consensus.cc:348] T 7993f9c3e2a44f408bbf5d1e395e2306 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:51.617769 24667 raft_consensus.cc:686] T 442d4a29b1d743599f4b6b4871e26668 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 LEADER]: Becoming Leader. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:03:51.600072 24458 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:55552 (request call id 56) took 1173 ms. Trace:
I20260925 03:03:51.601372 24456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 100c113a9fd84cc8adf2ec7b33ea0093. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:03:51.618311 24458 rpcz_store.cc:276] 0925 03:03:50.426512 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:03:51.474253 (+1047741us) service_pool.cc:227] Handling call
0925 03:03:51.475953 (+ 1700us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0925 03:03:51.476053 (+ 100us) ts_tablet_manager.cc:676] Creating new metadata...
0925 03:03:51.594185 (+118132us) tablet_metadata.cc:641] Metadata flushed
0925 03:03:51.599584 (+ 5399us) tablet_replica.cc:205] Creating consensus instance
0925 03:03:51.600016 (+ 432us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:03:51.618717 24667 consensus_queue.cc:237] T 442d4a29b1d743599f4b6b4871e26668 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:51.600852 24663 ts_tablet_manager.cc:1397] T 7993f9c3e2a44f408bbf5d1e395e2306 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent bootstrapping tablet: real 0.108s user 0.031s sys 0.022s
I20260925 03:03:51.600967 24461 rpcz_store.cc:276] 0925 03:03:50.328252 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:03:51.422871 (+1094619us) service_pool.cc:227] Handling call
0925 03:03:51.435108 (+ 12237us) spinlock_profiling.cc:243] Waited 10.3 ms on lock 0x7b5800001598. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f43733ecf11 00007f4373570851 00007f4373570702 00007f437c460b09 00007f437b39c730 00007f437b47ed21 00007f437b45c1e3 00007f437b40d869 00007f4374a0fc44 00007f4374a0fbd2 00007f4374a0fb01
0925 03:03:51.436441 (+ 1333us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0925 03:03:51.436622 (+ 181us) ts_tablet_manager.cc:676] Creating new metadata...
0925 03:03:51.523809 (+ 87187us) tablet_metadata.cc:641] Metadata flushed
0925 03:03:51.600183 (+ 76374us) tablet_replica.cc:205] Creating consensus instance
0925 03:03:51.600522 (+ 339us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":23722240}
I20260925 03:03:51.622692 24663 raft_consensus.cc:348] T 7993f9c3e2a44f408bbf5d1e395e2306 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:51.623541 24663 raft_consensus.cc:374] T 7993f9c3e2a44f408bbf5d1e395e2306 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:51.623852 24663 raft_consensus.cc:729] T 7993f9c3e2a44f408bbf5d1e395e2306 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Initialized, Role: FOLLOWER
I20260925 03:03:51.624593 24663 consensus_queue.cc:260] T 7993f9c3e2a44f408bbf5d1e395e2306 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:03:51.625093 24574 tablet.cc:2369] T b1e85248d3b04b979dd080caeeb02fdb P a04a37fc4aa7460cbfdbeeb4e451e1f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 03:03:51.617877 24664 raft_consensus.cc:374] T 7993f9c3e2a44f408bbf5d1e395e2306 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:51.627197 24664 raft_consensus.cc:729] T 7993f9c3e2a44f408bbf5d1e395e2306 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Initialized, Role: FOLLOWER
I20260925 03:03:51.628006 24664 consensus_queue.cc:260] T 7993f9c3e2a44f408bbf5d1e395e2306 P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:51.629776 24662 ts_tablet_manager.cc:1428] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c: Time spent starting tablet: real 0.033s user 0.005s sys 0.001s
I20260925 03:03:51.630347 24664 ts_tablet_manager.cc:1428] T 7993f9c3e2a44f408bbf5d1e395e2306 P 869b0a8575b54d08ae2817fad5dd42d5: Time spent starting tablet: real 0.016s user 0.007s sys 0.000s
I20260925 03:03:51.631368 24664 tablet_bootstrap.cc:492] T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5: Bootstrap starting.
I20260925 03:03:51.633082 24663 ts_tablet_manager.cc:1428] T 7993f9c3e2a44f408bbf5d1e395e2306 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent starting tablet: real 0.013s user 0.007s sys 0.000s
I20260925 03:03:51.637877 24663 tablet_bootstrap.cc:492] T edcf6e6a422d4d198d4fbb0f7d3d845d P a04a37fc4aa7460cbfdbeeb4e451e1f5: Bootstrap starting.
I20260925 03:03:51.630779 24662 tablet_bootstrap.cc:492] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c: Bootstrap starting.
I20260925 03:03:51.643787 24664 tablet_bootstrap.cc:654] T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:51.643818 24663 tablet_bootstrap.cc:654] T edcf6e6a422d4d198d4fbb0f7d3d845d P a04a37fc4aa7460cbfdbeeb4e451e1f5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:51.643646 24369 catalog_manager.cc:5495] T 442d4a29b1d743599f4b6b4871e26668 P a04a37fc4aa7460cbfdbeeb4e451e1f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2). New cstate: current_term: 1 leader_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: UNKNOWN } } }
I20260925 03:03:51.649467 24662 tablet_bootstrap.cc:654] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:51.635428 24456 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:55552 (request call id 58) took 1160 ms. Trace:
I20260925 03:03:51.651399 24456 rpcz_store.cc:276] 0925 03:03:50.474690 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:03:51.545455 (+1070765us) service_pool.cc:227] Handling call
0925 03:03:51.600844 (+ 55389us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0925 03:03:51.600954 (+ 110us) ts_tablet_manager.cc:676] Creating new metadata...
0925 03:03:51.627516 (+ 26562us) tablet_metadata.cc:641] Metadata flushed
0925 03:03:51.634902 (+ 7386us) tablet_replica.cc:205] Creating consensus instance
0925 03:03:51.635352 (+ 450us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:03:51.662390 24664 tablet_bootstrap.cc:492] T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5: No bootstrap required, opened a new log
I20260925 03:03:51.662791 24662 tablet_bootstrap.cc:492] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c: No bootstrap required, opened a new log
I20260925 03:03:51.662917 24664 ts_tablet_manager.cc:1397] T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5: Time spent bootstrapping tablet: real 0.032s user 0.006s sys 0.007s
I20260925 03:03:51.663288 24662 ts_tablet_manager.cc:1397] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c: Time spent bootstrapping tablet: real 0.033s user 0.013s sys 0.004s
I20260925 03:03:51.663455 24663 tablet_bootstrap.cc:492] T edcf6e6a422d4d198d4fbb0f7d3d845d P a04a37fc4aa7460cbfdbeeb4e451e1f5: No bootstrap required, opened a new log
I20260925 03:03:51.664088 24663 ts_tablet_manager.cc:1397] T edcf6e6a422d4d198d4fbb0f7d3d845d P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent bootstrapping tablet: real 0.026s user 0.013s sys 0.007s
I20260925 03:03:51.665880 24664 raft_consensus.cc:348] T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:51.666769 24664 raft_consensus.cc:374] T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:51.666587 24662 raft_consensus.cc:348] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:51.667140 24664 raft_consensus.cc:729] T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Initialized, Role: FOLLOWER
I20260925 03:03:51.667394 24662 raft_consensus.cc:374] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:51.667136 24663 raft_consensus.cc:348] T edcf6e6a422d4d198d4fbb0f7d3d845d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:51.667802 24662 raft_consensus.cc:729] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Initialized, Role: FOLLOWER
I20260925 03:03:51.668253 24663 raft_consensus.cc:374] T edcf6e6a422d4d198d4fbb0f7d3d845d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:51.668800 24663 raft_consensus.cc:729] T edcf6e6a422d4d198d4fbb0f7d3d845d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Initialized, Role: FOLLOWER
I20260925 03:03:51.668197 24664 consensus_queue.cc:260] T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:51.668646 24662 consensus_queue.cc:260] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:51.669780 24663 consensus_queue.cc:260] T edcf6e6a422d4d198d4fbb0f7d3d845d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:51.671056 24664 ts_tablet_manager.cc:1428] T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260925 03:03:51.672266 24664 tablet_bootstrap.cc:492] T edcf6e6a422d4d198d4fbb0f7d3d845d P 869b0a8575b54d08ae2817fad5dd42d5: Bootstrap starting.
I20260925 03:03:51.672425 24663 ts_tablet_manager.cc:1428] T edcf6e6a422d4d198d4fbb0f7d3d845d P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent starting tablet: real 0.008s user 0.000s sys 0.006s
I20260925 03:03:51.673496 24663 tablet_bootstrap.cc:492] T 48ca6ccd98194cf2af7b5278d15bf4ea P a04a37fc4aa7460cbfdbeeb4e451e1f5: Bootstrap starting.
I20260925 03:03:51.679149 24664 tablet_bootstrap.cc:654] T edcf6e6a422d4d198d4fbb0f7d3d845d P 869b0a8575b54d08ae2817fad5dd42d5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:51.679165 24663 tablet_bootstrap.cc:654] T 48ca6ccd98194cf2af7b5278d15bf4ea P a04a37fc4aa7460cbfdbeeb4e451e1f5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:51.671025 24662 ts_tablet_manager.cc:1428] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260925 03:03:51.688469 24662 tablet_bootstrap.cc:492] T 7993f9c3e2a44f408bbf5d1e395e2306 P eeab93abfdb141a895002ec9f312296c: Bootstrap starting.
I20260925 03:03:51.703233 24662 tablet_bootstrap.cc:654] T 7993f9c3e2a44f408bbf5d1e395e2306 P eeab93abfdb141a895002ec9f312296c: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:51.703519 24667 raft_consensus.cc:482] T b1e85248d3b04b979dd080caeeb02fdb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:51.703989 24667 raft_consensus.cc:504] T b1e85248d3b04b979dd080caeeb02fdb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:51.706066 24667 leader_election.cc:288] T b1e85248d3b04b979dd080caeeb02fdb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:51.707806 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1e85248d3b04b979dd080caeeb02fdb" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:03:51.707753 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1e85248d3b04b979dd080caeeb02fdb" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
W20260925 03:03:51.709177 24514 leader_election.cc:341] T b1e85248d3b04b979dd080caeeb02fdb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187): Illegal state: must be running to vote when last-logged opid is not known
W20260925 03:03:51.709650 24511 leader_election.cc:341] T b1e85248d3b04b979dd080caeeb02fdb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:51.710342 24511 leader_election.cc:302] T b1e85248d3b04b979dd080caeeb02fdb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:03:51.711491 24667 raft_consensus.cc:2726] T b1e85248d3b04b979dd080caeeb02fdb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 03:03:51.714217 24663 tablet_bootstrap.cc:492] T 48ca6ccd98194cf2af7b5278d15bf4ea P a04a37fc4aa7460cbfdbeeb4e451e1f5: No bootstrap required, opened a new log
I20260925 03:03:51.714702 24663 ts_tablet_manager.cc:1397] T 48ca6ccd98194cf2af7b5278d15bf4ea P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent bootstrapping tablet: real 0.041s user 0.015s sys 0.015s
I20260925 03:03:51.715176 24664 tablet_bootstrap.cc:492] T edcf6e6a422d4d198d4fbb0f7d3d845d P 869b0a8575b54d08ae2817fad5dd42d5: No bootstrap required, opened a new log
I20260925 03:03:51.715636 24664 ts_tablet_manager.cc:1397] T edcf6e6a422d4d198d4fbb0f7d3d845d P 869b0a8575b54d08ae2817fad5dd42d5: Time spent bootstrapping tablet: real 0.044s user 0.013s sys 0.027s
I20260925 03:03:51.715801 24662 tablet_bootstrap.cc:492] T 7993f9c3e2a44f408bbf5d1e395e2306 P eeab93abfdb141a895002ec9f312296c: No bootstrap required, opened a new log
I20260925 03:03:51.716200 24662 ts_tablet_manager.cc:1397] T 7993f9c3e2a44f408bbf5d1e395e2306 P eeab93abfdb141a895002ec9f312296c: Time spent bootstrapping tablet: real 0.028s user 0.008s sys 0.002s
I20260925 03:03:51.717458 24663 raft_consensus.cc:348] T 48ca6ccd98194cf2af7b5278d15bf4ea P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:51.718235 24664 raft_consensus.cc:348] T edcf6e6a422d4d198d4fbb0f7d3d845d P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:51.718240 24663 raft_consensus.cc:374] T 48ca6ccd98194cf2af7b5278d15bf4ea P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:51.718982 24664 raft_consensus.cc:374] T edcf6e6a422d4d198d4fbb0f7d3d845d P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:51.718631 24662 raft_consensus.cc:348] T 7993f9c3e2a44f408bbf5d1e395e2306 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:51.719161 24663 raft_consensus.cc:729] T 48ca6ccd98194cf2af7b5278d15bf4ea P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Initialized, Role: FOLLOWER
I20260925 03:03:51.719298 24664 raft_consensus.cc:729] T edcf6e6a422d4d198d4fbb0f7d3d845d P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Initialized, Role: FOLLOWER
I20260925 03:03:51.719429 24662 raft_consensus.cc:374] T 7993f9c3e2a44f408bbf5d1e395e2306 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:51.719892 24662 raft_consensus.cc:729] T 7993f9c3e2a44f408bbf5d1e395e2306 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Initialized, Role: FOLLOWER
I20260925 03:03:51.720009 24664 consensus_queue.cc:260] T edcf6e6a422d4d198d4fbb0f7d3d845d P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:51.720055 24663 consensus_queue.cc:260] T 48ca6ccd98194cf2af7b5278d15bf4ea P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:51.720671 24662 consensus_queue.cc:260] T 7993f9c3e2a44f408bbf5d1e395e2306 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:51.722357 24664 ts_tablet_manager.cc:1428] T edcf6e6a422d4d198d4fbb0f7d3d845d P 869b0a8575b54d08ae2817fad5dd42d5: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260925 03:03:51.723377 24664 tablet_bootstrap.cc:492] T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5: Bootstrap starting.
I20260925 03:03:51.723798 24662 ts_tablet_manager.cc:1428] T 7993f9c3e2a44f408bbf5d1e395e2306 P eeab93abfdb141a895002ec9f312296c: Time spent starting tablet: real 0.007s user 0.000s sys 0.007s
I20260925 03:03:51.725694 24663 ts_tablet_manager.cc:1428] T 48ca6ccd98194cf2af7b5278d15bf4ea P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent starting tablet: real 0.011s user 0.008s sys 0.000s
I20260925 03:03:51.726656 24663 tablet_bootstrap.cc:492] T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5: Bootstrap starting.
I20260925 03:03:51.728262 24671 raft_consensus.cc:482] T edcf6e6a422d4d198d4fbb0f7d3d845d P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:51.729023 24664 tablet_bootstrap.cc:654] T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:51.728745 24671 raft_consensus.cc:504] T edcf6e6a422d4d198d4fbb0f7d3d845d P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:51.730799 24671 leader_election.cc:288] T edcf6e6a422d4d198d4fbb0f7d3d845d P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:51.731858 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcf6e6a422d4d198d4fbb0f7d3d845d" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:03:51.732878 24663 tablet_bootstrap.cc:654] T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:51.732530 24622 raft_consensus.cc:2454] T edcf6e6a422d4d198d4fbb0f7d3d845d P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 0.
I20260925 03:03:51.733446 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcf6e6a422d4d198d4fbb0f7d3d845d" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:03:51.734045 24548 raft_consensus.cc:2454] T edcf6e6a422d4d198d4fbb0f7d3d845d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 0.
I20260925 03:03:51.734212 24436 leader_election.cc:302] T edcf6e6a422d4d198d4fbb0f7d3d845d P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:03:51.735095 24671 raft_consensus.cc:2781] T edcf6e6a422d4d198d4fbb0f7d3d845d P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 03:03:51.735423 24671 raft_consensus.cc:482] T edcf6e6a422d4d198d4fbb0f7d3d845d P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:03:51.735734 24671 raft_consensus.cc:3037] T edcf6e6a422d4d198d4fbb0f7d3d845d P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:51.735963 24662 tablet_bootstrap.cc:492] T 71f461a6e2ff434993b78a2d077f0abd P eeab93abfdb141a895002ec9f312296c: Bootstrap starting.
I20260925 03:03:51.742120 24662 tablet_bootstrap.cc:654] T 71f461a6e2ff434993b78a2d077f0abd P eeab93abfdb141a895002ec9f312296c: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:51.744249 24671 raft_consensus.cc:504] T edcf6e6a422d4d198d4fbb0f7d3d845d P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:51.747162 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcf6e6a422d4d198d4fbb0f7d3d845d" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:03:51.747767 24548 raft_consensus.cc:3037] T edcf6e6a422d4d198d4fbb0f7d3d845d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:51.748477 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcf6e6a422d4d198d4fbb0f7d3d845d" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:03:51.749121 24622 raft_consensus.cc:3037] T edcf6e6a422d4d198d4fbb0f7d3d845d P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:51.754936 24548 raft_consensus.cc:2454] T edcf6e6a422d4d198d4fbb0f7d3d845d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 1.
I20260925 03:03:51.756134 24437 leader_election.cc:302] T edcf6e6a422d4d198d4fbb0f7d3d845d P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:03:51.757027 24622 raft_consensus.cc:2454] T edcf6e6a422d4d198d4fbb0f7d3d845d P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 1.
I20260925 03:03:51.758896 24671 leader_election.cc:288] T edcf6e6a422d4d198d4fbb0f7d3d845d P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:51.759204 24663 tablet_bootstrap.cc:492] T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5: No bootstrap required, opened a new log
I20260925 03:03:51.759496 24671 raft_consensus.cc:2781] T edcf6e6a422d4d198d4fbb0f7d3d845d P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:03:51.760053 24671 raft_consensus.cc:686] T edcf6e6a422d4d198d4fbb0f7d3d845d P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 LEADER]: Becoming Leader. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:03:51.760123 24663 ts_tablet_manager.cc:1397] T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent bootstrapping tablet: real 0.034s user 0.013s sys 0.007s
I20260925 03:03:51.760991 24671 consensus_queue.cc:237] T edcf6e6a422d4d198d4fbb0f7d3d845d P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:51.763495 24663 raft_consensus.cc:348] T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } }
I20260925 03:03:51.764380 24663 raft_consensus.cc:374] T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:51.764746 24663 raft_consensus.cc:729] T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Initialized, Role: FOLLOWER
I20260925 03:03:51.765545 24663 consensus_queue.cc:260] T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } }
I20260925 03:03:51.767673 24663 ts_tablet_manager.cc:1428] T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260925 03:03:51.768217 24664 tablet_bootstrap.cc:492] T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5: No bootstrap required, opened a new log
I20260925 03:03:51.768781 24664 ts_tablet_manager.cc:1397] T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5: Time spent bootstrapping tablet: real 0.046s user 0.014s sys 0.000s
I20260925 03:03:51.768932 24663 tablet_bootstrap.cc:492] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5: Bootstrap starting.
I20260925 03:03:51.771610 24664 raft_consensus.cc:348] T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:51.774608 24663 tablet_bootstrap.cc:654] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:51.772467 24664 raft_consensus.cc:374] T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:51.775556 24664 raft_consensus.cc:729] T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Initialized, Role: FOLLOWER
I20260925 03:03:51.776266 24664 consensus_queue.cc:260] T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:51.777570 24662 tablet_bootstrap.cc:492] T 71f461a6e2ff434993b78a2d077f0abd P eeab93abfdb141a895002ec9f312296c: No bootstrap required, opened a new log
I20260925 03:03:51.778108 24662 ts_tablet_manager.cc:1397] T 71f461a6e2ff434993b78a2d077f0abd P eeab93abfdb141a895002ec9f312296c: Time spent bootstrapping tablet: real 0.042s user 0.019s sys 0.018s
I20260925 03:03:51.782562 24662 raft_consensus.cc:348] T 71f461a6e2ff434993b78a2d077f0abd P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:51.783339 24662 raft_consensus.cc:374] T 71f461a6e2ff434993b78a2d077f0abd P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:51.783615 24662 raft_consensus.cc:729] T 71f461a6e2ff434993b78a2d077f0abd P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Initialized, Role: FOLLOWER
I20260925 03:03:51.784402 24662 consensus_queue.cc:260] T 71f461a6e2ff434993b78a2d077f0abd P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:51.786702 24662 ts_tablet_manager.cc:1428] T 71f461a6e2ff434993b78a2d077f0abd P eeab93abfdb141a895002ec9f312296c: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20260925 03:03:51.787712 24662 tablet_bootstrap.cc:492] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c: Bootstrap starting.
I20260925 03:03:51.789530 24663 tablet_bootstrap.cc:492] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5: No bootstrap required, opened a new log
I20260925 03:03:51.790038 24663 ts_tablet_manager.cc:1397] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent bootstrapping tablet: real 0.021s user 0.015s sys 0.005s
I20260925 03:03:51.791956 24664 ts_tablet_manager.cc:1428] T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5: Time spent starting tablet: real 0.023s user 0.005s sys 0.002s
I20260925 03:03:51.792976 24664 tablet_bootstrap.cc:492] T 04e068a0f80941dc9138cee063f53bab P 869b0a8575b54d08ae2817fad5dd42d5: Bootstrap starting.
I20260925 03:03:51.792572 24663 raft_consensus.cc:348] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:51.793439 24663 raft_consensus.cc:374] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:51.793546 24662 tablet_bootstrap.cc:654] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:51.793823 24663 raft_consensus.cc:729] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Initialized, Role: FOLLOWER
I20260925 03:03:51.794649 24663 consensus_queue.cc:260] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:51.793797 24369 catalog_manager.cc:5495] T edcf6e6a422d4d198d4fbb0f7d3d845d P 869b0a8575b54d08ae2817fad5dd42d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1). New cstate: current_term: 1 leader_uuid: "869b0a8575b54d08ae2817fad5dd42d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: UNKNOWN } } }
I20260925 03:03:51.800239 24663 ts_tablet_manager.cc:1428] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent starting tablet: real 0.010s user 0.004s sys 0.002s
I20260925 03:03:51.801195 24663 tablet_bootstrap.cc:492] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Bootstrap starting.
I20260925 03:03:51.806313 24663 tablet_bootstrap.cc:654] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:51.809417 24664 tablet_bootstrap.cc:654] T 04e068a0f80941dc9138cee063f53bab P 869b0a8575b54d08ae2817fad5dd42d5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:51.817852 24663 tablet_bootstrap.cc:492] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5: No bootstrap required, opened a new log
I20260925 03:03:51.818466 24663 ts_tablet_manager.cc:1397] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent bootstrapping tablet: real 0.017s user 0.012s sys 0.004s
I20260925 03:03:51.821095 24663 raft_consensus.cc:348] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:51.822053 24663 raft_consensus.cc:374] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:51.822376 24663 raft_consensus.cc:729] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Initialized, Role: FOLLOWER
I20260925 03:03:51.823200 24663 consensus_queue.cc:260] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:51.823704 24670 raft_consensus.cc:482] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:51.824764 24670 raft_consensus.cc:504] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:51.825873 24663 ts_tablet_manager.cc:1428] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260925 03:03:51.826764 24663 tablet_bootstrap.cc:492] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Bootstrap starting.
I20260925 03:03:51.826853 24670 leader_election.cc:288] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:03:51.827092 24664 tablet_bootstrap.cc:492] T 04e068a0f80941dc9138cee063f53bab P 869b0a8575b54d08ae2817fad5dd42d5: No bootstrap required, opened a new log
I20260925 03:03:51.827675 24664 ts_tablet_manager.cc:1397] T 04e068a0f80941dc9138cee063f53bab P 869b0a8575b54d08ae2817fad5dd42d5: Time spent bootstrapping tablet: real 0.035s user 0.013s sys 0.010s
I20260925 03:03:51.828477 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f68804eb5a244d87bb395d50683c6012" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:03:51.828769 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f68804eb5a244d87bb395d50683c6012" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:03:51.829447 24548 raft_consensus.cc:2454] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 0.
W20260925 03:03:51.829777 24588 leader_election.cc:341] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:51.831638 24586 leader_election.cc:302] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:03:51.832090 24663 tablet_bootstrap.cc:654] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:51.832579 24662 tablet_bootstrap.cc:492] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c: No bootstrap required, opened a new log
I20260925 03:03:51.832792 24670 raft_consensus.cc:2781] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 03:03:51.833045 24662 ts_tablet_manager.cc:1397] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c: Time spent bootstrapping tablet: real 0.046s user 0.026s sys 0.015s
I20260925 03:03:51.833163 24670 raft_consensus.cc:482] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:03:51.833513 24670 raft_consensus.cc:3037] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:51.835549 24662 raft_consensus.cc:348] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:51.836421 24662 raft_consensus.cc:374] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:51.836661 24662 raft_consensus.cc:729] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Initialized, Role: FOLLOWER
I20260925 03:03:51.837496 24662 consensus_queue.cc:260] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:51.839865 24667 raft_consensus.cc:482] T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:51.840373 24667 raft_consensus.cc:504] T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } }
I20260925 03:03:51.841037 24670 raft_consensus.cc:504] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:51.842640 24667 leader_election.cc:288] T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187)
I20260925 03:03:51.842478 24664 raft_consensus.cc:348] T 04e068a0f80941dc9138cee063f53bab P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } }
I20260925 03:03:51.843107 24664 raft_consensus.cc:374] T 04e068a0f80941dc9138cee063f53bab P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:51.843451 24664 raft_consensus.cc:729] T 04e068a0f80941dc9138cee063f53bab P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Initialized, Role: FOLLOWER
I20260925 03:03:51.843693 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f68804eb5a244d87bb395d50683c6012" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:03:51.844031 24664 consensus_queue.cc:260] T 04e068a0f80941dc9138cee063f53bab P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } }
I20260925 03:03:51.844573 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e068a0f80941dc9138cee063f53bab" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
W20260925 03:03:51.845669 24588 leader_election.cc:341] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:51.846201 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f68804eb5a244d87bb395d50683c6012" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:03:51.846732 24548 raft_consensus.cc:3037] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:51.847322 24472 raft_consensus.cc:2454] T 04e068a0f80941dc9138cee063f53bab P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 0.
I20260925 03:03:51.846832 24664 ts_tablet_manager.cc:1428] T 04e068a0f80941dc9138cee063f53bab P 869b0a8575b54d08ae2817fad5dd42d5: Time spent starting tablet: real 0.019s user 0.002s sys 0.002s
I20260925 03:03:51.849747 24664 tablet_bootstrap.cc:492] T 293ef69caf514ac98510a68495f54eda P 869b0a8575b54d08ae2817fad5dd42d5: Bootstrap starting.
I20260925 03:03:51.850770 24514 leader_election.cc:302] T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:03:51.852367 24667 raft_consensus.cc:2781] T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 03:03:51.846756 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e068a0f80941dc9138cee063f53bab" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:03:51.853421 24622 raft_consensus.cc:2454] T 04e068a0f80941dc9138cee063f53bab P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 0.
I20260925 03:03:51.855068 24667 raft_consensus.cc:482] T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:03:51.855809 24667 raft_consensus.cc:3037] T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:51.860136 24716 raft_consensus.cc:482] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:51.860621 24716 raft_consensus.cc:504] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:51.858443 24548 raft_consensus.cc:2454] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 1.
I20260925 03:03:51.863821 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e92f377a79354a6ebb4eec67303c13e3" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:03:51.864543 24586 leader_election.cc:302] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:03:51.865108 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e92f377a79354a6ebb4eec67303c13e3" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:03:51.865731 24472 raft_consensus.cc:2454] T e92f377a79354a6ebb4eec67303c13e3 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 0.
W20260925 03:03:51.866302 24586 leader_election.cc:341] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:51.866255 24667 raft_consensus.cc:504] T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } }
I20260925 03:03:51.866496 24717 raft_consensus.cc:2781] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:03:51.859735 24662 ts_tablet_manager.cc:1428] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c: Time spent starting tablet: real 0.026s user 0.011s sys 0.000s
I20260925 03:03:51.867046 24588 leader_election.cc:302] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:03:51.858528 24670 leader_election.cc:288] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 1 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:03:51.856073 24664 tablet_bootstrap.cc:654] T 293ef69caf514ac98510a68495f54eda P 869b0a8575b54d08ae2817fad5dd42d5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:51.869308 24670 raft_consensus.cc:2781] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 03:03:51.869673 24670 raft_consensus.cc:482] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:03:51.869576 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e068a0f80941dc9138cee063f53bab" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:03:51.870080 24670 raft_consensus.cc:3037] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:51.870437 24472 raft_consensus.cc:3037] T 04e068a0f80941dc9138cee063f53bab P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:51.868253 24662 tablet_bootstrap.cc:492] T b1e85248d3b04b979dd080caeeb02fdb P eeab93abfdb141a895002ec9f312296c: Bootstrap starting.
I20260925 03:03:51.874058 24664 tablet_bootstrap.cc:492] T 293ef69caf514ac98510a68495f54eda P 869b0a8575b54d08ae2817fad5dd42d5: No bootstrap required, opened a new log
I20260925 03:03:51.874511 24663 tablet_bootstrap.cc:492] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5: No bootstrap required, opened a new log
I20260925 03:03:51.875058 24663 ts_tablet_manager.cc:1397] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent bootstrapping tablet: real 0.048s user 0.015s sys 0.012s
I20260925 03:03:51.875164 24711 raft_consensus.cc:482] T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:51.875605 24711 raft_consensus.cc:504] T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:51.877723 24662 tablet_bootstrap.cc:654] T b1e85248d3b04b979dd080caeeb02fdb P eeab93abfdb141a895002ec9f312296c: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:51.877455 24663 raft_consensus.cc:348] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:51.877761 24711 leader_election.cc:288] T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:51.878190 24663 raft_consensus.cc:374] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:51.878459 24663 raft_consensus.cc:729] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Initialized, Role: FOLLOWER
I20260925 03:03:51.879135 24663 consensus_queue.cc:260] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:51.879424 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48ca6ccd98194cf2af7b5278d15bf4ea" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:03:51.880488 24716 leader_election.cc:288] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:03:51.880616 24663 ts_tablet_manager.cc:1428] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
W20260925 03:03:51.880771 24436 leader_election.cc:341] T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:51.881526 24663 tablet_bootstrap.cc:492] T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Bootstrap starting.
I20260925 03:03:51.874513 24664 ts_tablet_manager.cc:1397] T 293ef69caf514ac98510a68495f54eda P 869b0a8575b54d08ae2817fad5dd42d5: Time spent bootstrapping tablet: real 0.025s user 0.009s sys 0.001s
I20260925 03:03:51.882292 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48ca6ccd98194cf2af7b5278d15bf4ea" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:03:51.882906 24548 raft_consensus.cc:2454] T 48ca6ccd98194cf2af7b5278d15bf4ea P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 0.
I20260925 03:03:51.883956 24437 leader_election.cc:302] T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:03:51.884212 24664 raft_consensus.cc:348] T 293ef69caf514ac98510a68495f54eda P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:51.884927 24664 raft_consensus.cc:374] T 293ef69caf514ac98510a68495f54eda P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:51.885042 24711 raft_consensus.cc:2781] T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 03:03:51.885200 24664 raft_consensus.cc:729] T 293ef69caf514ac98510a68495f54eda P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Initialized, Role: FOLLOWER
I20260925 03:03:51.885457 24711 raft_consensus.cc:482] T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:03:51.885848 24711 raft_consensus.cc:3037] T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:51.886439 24663 tablet_bootstrap.cc:654] T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:51.886878 24664 consensus_queue.cc:260] T 293ef69caf514ac98510a68495f54eda P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:51.868268 24667 leader_election.cc:288] T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 election: Requested vote from peers eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187)
I20260925 03:03:51.895937 24664 ts_tablet_manager.cc:1428] T 293ef69caf514ac98510a68495f54eda P 869b0a8575b54d08ae2817fad5dd42d5: Time spent starting tablet: real 0.014s user 0.005s sys 0.000s
I20260925 03:03:51.897234 24664 tablet_bootstrap.cc:492] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5: Bootstrap starting.
I20260925 03:03:51.902560 24664 tablet_bootstrap.cc:654] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:51.909014 24716 raft_consensus.cc:482] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:51.909581 24716 raft_consensus.cc:504] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:51.913200 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f22ddd7d1c444dc58f45a1fb0b3e02bc" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:03:51.913980 24548 raft_consensus.cc:2454] T f22ddd7d1c444dc58f45a1fb0b3e02bc P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 0.
I20260925 03:03:51.915475 24586 leader_election.cc:302] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:03:51.918900 24662 tablet_bootstrap.cc:492] T b1e85248d3b04b979dd080caeeb02fdb P eeab93abfdb141a895002ec9f312296c: No bootstrap required, opened a new log
I20260925 03:03:51.919242 24716 leader_election.cc:288] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:03:51.921036 24716 raft_consensus.cc:2781] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 03:03:51.921501 24716 raft_consensus.cc:482] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:03:51.922173 24716 raft_consensus.cc:3037] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:51.919381 24662 ts_tablet_manager.cc:1397] T b1e85248d3b04b979dd080caeeb02fdb P eeab93abfdb141a895002ec9f312296c: Time spent bootstrapping tablet: real 0.051s user 0.007s sys 0.009s
I20260925 03:03:51.921586 24711 raft_consensus.cc:504] T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:51.924957 24662 raft_consensus.cc:348] T b1e85248d3b04b979dd080caeeb02fdb P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:51.925203 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48ca6ccd98194cf2af7b5278d15bf4ea" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:03:51.925110 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48ca6ccd98194cf2af7b5278d15bf4ea" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:03:51.925976 24548 raft_consensus.cc:3037] T 48ca6ccd98194cf2af7b5278d15bf4ea P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:51.927145 24662 raft_consensus.cc:374] T b1e85248d3b04b979dd080caeeb02fdb P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:51.927597 24662 raft_consensus.cc:729] T b1e85248d3b04b979dd080caeeb02fdb P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Initialized, Role: FOLLOWER
I20260925 03:03:51.928431 24662 consensus_queue.cc:260] T b1e85248d3b04b979dd080caeeb02fdb P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:03:51.926954 24436 leader_election.cc:341] T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:51.932294 24662 ts_tablet_manager.cc:1428] T b1e85248d3b04b979dd080caeeb02fdb P eeab93abfdb141a895002ec9f312296c: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20260925 03:03:51.903290 24670 raft_consensus.cc:504] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:51.936430 24667 raft_consensus.cc:482] T 7993f9c3e2a44f408bbf5d1e395e2306 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:51.936673 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e92f377a79354a6ebb4eec67303c13e3" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:03:51.936952 24667 raft_consensus.cc:504] T 7993f9c3e2a44f408bbf5d1e395e2306 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:51.937582 24693 raft_consensus.cc:482] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:51.939311 24662 tablet_bootstrap.cc:492] T 48ca6ccd98194cf2af7b5278d15bf4ea P eeab93abfdb141a895002ec9f312296c: Bootstrap starting.
I20260925 03:03:51.940078 24717 raft_consensus.cc:686] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c [term 1 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:03:51.940991 24667 leader_election.cc:288] T 7993f9c3e2a44f408bbf5d1e395e2306 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
W20260925 03:03:51.938616 24586 leader_election.cc:341] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:51.938155 24693 raft_consensus.cc:504] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:51.942349 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7993f9c3e2a44f408bbf5d1e395e2306" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:03:51.940181 24472 raft_consensus.cc:2454] T 04e068a0f80941dc9138cee063f53bab P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 1.
I20260925 03:03:51.943406 24473 raft_consensus.cc:2454] T 7993f9c3e2a44f408bbf5d1e395e2306 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 0.
I20260925 03:03:51.944913 24548 raft_consensus.cc:2454] T 48ca6ccd98194cf2af7b5278d15bf4ea P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 1.
I20260925 03:03:51.945773 24514 leader_election.cc:302] T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:03:51.946372 24693 leader_election.cc:288] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
W20260925 03:03:51.947454 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 80784.3us
I20260925 03:03:51.948115 24437 leader_election.cc:302] T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:03:51.939148 24711 leader_election.cc:288] T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:51.942857 24670 leader_election.cc:288] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 1 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:03:51.949482 24671 raft_consensus.cc:2781] T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:03:51.949949 24693 raft_consensus.cc:2781] T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:03:51.951540 24671 raft_consensus.cc:686] T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 LEADER]: Becoming Leader. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:03:51.948247 24514 leader_election.cc:302] T 7993f9c3e2a44f408bbf5d1e395e2306 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:03:51.952415 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "293ef69caf514ac98510a68495f54eda" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:03:51.951541 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f22ddd7d1c444dc58f45a1fb0b3e02bc" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:03:51.953424 24472 raft_consensus.cc:2454] T 293ef69caf514ac98510a68495f54eda P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 0.
I20260925 03:03:51.954627 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e068a0f80941dc9138cee063f53bab" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:03:51.954478 24671 consensus_queue.cc:237] T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:51.955272 24622 raft_consensus.cc:3037] T 04e068a0f80941dc9138cee063f53bab P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:51.953442 24667 raft_consensus.cc:2781] T 7993f9c3e2a44f408bbf5d1e395e2306 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 03:03:51.956070 24667 raft_consensus.cc:482] T 7993f9c3e2a44f408bbf5d1e395e2306 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:03:51.956408 24667 raft_consensus.cc:3037] T 7993f9c3e2a44f408bbf5d1e395e2306 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:51.954953 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e92f377a79354a6ebb4eec67303c13e3" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:03:51.956956 24471 raft_consensus.cc:3037] T e92f377a79354a6ebb4eec67303c13e3 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:51.957779 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7993f9c3e2a44f408bbf5d1e395e2306" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:03:51.958470 24473 raft_consensus.cc:2454] T f22ddd7d1c444dc58f45a1fb0b3e02bc P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 0.
I20260925 03:03:51.941957 24717 consensus_queue.cc:237] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:51.959728 24621 raft_consensus.cc:2454] T 7993f9c3e2a44f408bbf5d1e395e2306 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 0.
I20260925 03:03:51.962208 24663 tablet_bootstrap.cc:492] T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5: No bootstrap required, opened a new log
I20260925 03:03:51.962675 24663 ts_tablet_manager.cc:1397] T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent bootstrapping tablet: real 0.081s user 0.018s sys 0.004s
I20260925 03:03:51.964497 24514 leader_election.cc:302] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:03:51.969420 24663 raft_consensus.cc:348] T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:51.970268 24663 raft_consensus.cc:374] T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:51.970531 24663 raft_consensus.cc:729] T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Initialized, Role: FOLLOWER
I20260925 03:03:51.975795 24662 tablet_bootstrap.cc:654] T 48ca6ccd98194cf2af7b5278d15bf4ea P eeab93abfdb141a895002ec9f312296c: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:51.978631 24622 raft_consensus.cc:2454] T 04e068a0f80941dc9138cee063f53bab P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 1.
I20260925 03:03:51.971199 24663 consensus_queue.cc:260] T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:51.977722 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "293ef69caf514ac98510a68495f54eda" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:03:51.975944 24471 raft_consensus.cc:2454] T e92f377a79354a6ebb4eec67303c13e3 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 1.
I20260925 03:03:51.981449 24723 raft_consensus.cc:2781] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 03:03:51.979439 24693 raft_consensus.cc:686] T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 LEADER]: Becoming Leader. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:03:51.986657 24693 consensus_queue.cc:237] T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } }
I20260925 03:03:51.995275 24716 raft_consensus.cc:504] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:51.996513 24369 catalog_manager.cc:5495] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c reported cstate change: term changed from 0 to 1, leader changed from <none> to eeab93abfdb141a895002ec9f312296c (127.22.193.3). New cstate: current_term: 1 leader_uuid: "eeab93abfdb141a895002ec9f312296c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: HEALTHY } } }
I20260925 03:03:51.998415 24716 leader_election.cc:288] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 1 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:03:51.998724 24664 tablet_bootstrap.cc:492] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5: No bootstrap required, opened a new log
I20260925 03:03:51.998590 24667 raft_consensus.cc:504] T 7993f9c3e2a44f408bbf5d1e395e2306 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:51.999176 24664 ts_tablet_manager.cc:1397] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5: Time spent bootstrapping tablet: real 0.102s user 0.006s sys 0.006s
I20260925 03:03:52.000658 24663 ts_tablet_manager.cc:1428] T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent starting tablet: real 0.038s user 0.004s sys 0.006s
I20260925 03:03:52.001595 24663 tablet_bootstrap.cc:492] T 0e54ca1757ee4e42956a9cebbeca4f62 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Bootstrap starting.
I20260925 03:03:52.002483 24664 raft_consensus.cc:348] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:52.003197 24664 raft_consensus.cc:374] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:52.003580 24664 raft_consensus.cc:729] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Initialized, Role: FOLLOWER
I20260925 03:03:52.004606 24664 consensus_queue.cc:260] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:52.006330 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f22ddd7d1c444dc58f45a1fb0b3e02bc" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:03:52.006946 24548 raft_consensus.cc:3037] T f22ddd7d1c444dc58f45a1fb0b3e02bc P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:52.007280 24664 ts_tablet_manager.cc:1428] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5: Time spent starting tablet: real 0.008s user 0.005s sys 0.003s
I20260925 03:03:52.008426 24664 tablet_bootstrap.cc:492] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5: Bootstrap starting.
I20260925 03:03:51.982769 24723 raft_consensus.cc:482] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:03:52.008996 24723 raft_consensus.cc:3037] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:52.013341 24588 leader_election.cc:302] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:03:52.015702 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f22ddd7d1c444dc58f45a1fb0b3e02bc" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:03:52.016335 24471 raft_consensus.cc:3037] T f22ddd7d1c444dc58f45a1fb0b3e02bc P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:52.020592 24716 raft_consensus.cc:2781] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:03:52.020431 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7993f9c3e2a44f408bbf5d1e395e2306" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:03:52.021566 24473 raft_consensus.cc:3037] T 7993f9c3e2a44f408bbf5d1e395e2306 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:52.021452 24716 raft_consensus.cc:686] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [term 1 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
W20260925 03:03:52.024214 24511 leader_election.cc:341] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:52.016459 24667 leader_election.cc:288] T 7993f9c3e2a44f408bbf5d1e395e2306 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:52.030378 24716 consensus_queue.cc:237] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:52.033927 24723 raft_consensus.cc:504] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:52.036784 24723 leader_election.cc:288] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:52.038386 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7993f9c3e2a44f408bbf5d1e395e2306" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:03:52.039408 24622 raft_consensus.cc:3037] T 7993f9c3e2a44f408bbf5d1e395e2306 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:52.050899 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "293ef69caf514ac98510a68495f54eda" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:03:52.028478 24471 raft_consensus.cc:2454] T f22ddd7d1c444dc58f45a1fb0b3e02bc P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 1.
I20260925 03:03:52.051891 24472 raft_consensus.cc:3037] T 293ef69caf514ac98510a68495f54eda P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:52.054251 24664 tablet_bootstrap.cc:654] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:52.057593 24671 raft_consensus.cc:482] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:52.058002 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "293ef69caf514ac98510a68495f54eda" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:03:52.058041 24671 raft_consensus.cc:504] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:52.058405 24723 consensus_queue.cc:1035] T 442d4a29b1d743599f4b6b4871e26668 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260925 03:03:52.059911 24671 leader_election.cc:288] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:52.060989 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f68804eb5a244d87bb395d50683c6012" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:03:52.061285 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f68804eb5a244d87bb395d50683c6012" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:03:52.061972 24547 raft_consensus.cc:2379] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 in current term 1: Already voted for candidate eeab93abfdb141a895002ec9f312296c in this term.
I20260925 03:03:52.062858 24473 raft_consensus.cc:2454] T 7993f9c3e2a44f408bbf5d1e395e2306 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 1.
I20260925 03:03:52.063319 24437 leader_election.cc:302] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:03:52.063411 24588 leader_election.cc:302] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:03:52.049501 24369 catalog_manager.cc:5495] T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1). New cstate: current_term: 1 leader_uuid: "869b0a8575b54d08ae2817fad5dd42d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: UNKNOWN } } }
W20260925 03:03:52.064774 24511 leader_election.cc:341] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:52.065025 24717 raft_consensus.cc:2781] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:03:52.065943 24717 raft_consensus.cc:686] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c [term 1 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:03:52.051561 24622 raft_consensus.cc:2454] T 7993f9c3e2a44f408bbf5d1e395e2306 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 1.
I20260925 03:03:52.067203 24671 raft_consensus.cc:3037] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:52.064802 24514 leader_election.cc:302] T 7993f9c3e2a44f408bbf5d1e395e2306 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:03:52.072845 24711 raft_consensus.cc:482] T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:52.073335 24662 tablet_bootstrap.cc:492] T 48ca6ccd98194cf2af7b5278d15bf4ea P eeab93abfdb141a895002ec9f312296c: No bootstrap required, opened a new log
I20260925 03:03:52.073331 24711 raft_consensus.cc:504] T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:52.073871 24662 ts_tablet_manager.cc:1397] T 48ca6ccd98194cf2af7b5278d15bf4ea P eeab93abfdb141a895002ec9f312296c: Time spent bootstrapping tablet: real 0.140s user 0.016s sys 0.011s
I20260925 03:03:52.075994 24711 leader_election.cc:288] T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:52.076927 24662 raft_consensus.cc:348] T 48ca6ccd98194cf2af7b5278d15bf4ea P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:52.077564 24662 raft_consensus.cc:374] T 48ca6ccd98194cf2af7b5278d15bf4ea P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:52.077826 24662 raft_consensus.cc:729] T 48ca6ccd98194cf2af7b5278d15bf4ea P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Initialized, Role: FOLLOWER
I20260925 03:03:52.077643 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1e85248d3b04b979dd080caeeb02fdb" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:03:52.078428 24662 consensus_queue.cc:260] T 48ca6ccd98194cf2af7b5278d15bf4ea P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:52.078688 24547 raft_consensus.cc:2454] T b1e85248d3b04b979dd080caeeb02fdb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 0.
I20260925 03:03:52.078418 24717 consensus_queue.cc:237] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:52.080920 24662 ts_tablet_manager.cc:1428] T 48ca6ccd98194cf2af7b5278d15bf4ea P eeab93abfdb141a895002ec9f312296c: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260925 03:03:52.081717 24662 tablet_bootstrap.cc:492] T 293ef69caf514ac98510a68495f54eda P eeab93abfdb141a895002ec9f312296c: Bootstrap starting.
I20260925 03:03:52.082345 24663 tablet_bootstrap.cc:654] T 0e54ca1757ee4e42956a9cebbeca4f62 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:52.080744 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1e85248d3b04b979dd080caeeb02fdb" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:03:52.082831 24472 raft_consensus.cc:2454] T 293ef69caf514ac98510a68495f54eda P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 1.
I20260925 03:03:52.088469 24371 catalog_manager.cc:5495] T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2). New cstate: current_term: 1 leader_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: UNKNOWN } } }
I20260925 03:03:52.085682 24437 leader_election.cc:302] T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:03:52.083864 24693 raft_consensus.cc:2781] T 7993f9c3e2a44f408bbf5d1e395e2306 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:03:52.091914 24514 leader_election.cc:302] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:03:52.085785 24620 raft_consensus.cc:2454] T b1e85248d3b04b979dd080caeeb02fdb P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 0.
I20260925 03:03:52.093040 24723 raft_consensus.cc:2781] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:03:52.093185 24693 raft_consensus.cc:686] T 7993f9c3e2a44f408bbf5d1e395e2306 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 LEADER]: Becoming Leader. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:03:52.094930 24693 consensus_queue.cc:237] T 7993f9c3e2a44f408bbf5d1e395e2306 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:52.093497 24723 raft_consensus.cc:686] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 LEADER]: Becoming Leader. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:03:52.102218 24723 consensus_queue.cc:237] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:52.091356 24711 raft_consensus.cc:2781] T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 03:03:52.106235 24711 raft_consensus.cc:482] T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:03:52.106737 24711 raft_consensus.cc:3037] T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:52.096119 24370 catalog_manager.cc:5495] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c reported cstate change: term changed from 0 to 1, leader changed from <none> to eeab93abfdb141a895002ec9f312296c (127.22.193.3). New cstate: current_term: 1 leader_uuid: "eeab93abfdb141a895002ec9f312296c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: HEALTHY } } }
I20260925 03:03:52.108522 24667 consensus_queue.cc:1035] T 442d4a29b1d743599f4b6b4871e26668 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
I20260925 03:03:52.121537 24693 raft_consensus.cc:482] T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:52.132875 24671 raft_consensus.cc:2726] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 03:03:52.139739 24662 tablet_bootstrap.cc:654] T 293ef69caf514ac98510a68495f54eda P eeab93abfdb141a895002ec9f312296c: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:52.136162 24693 raft_consensus.cc:504] T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:52.147132 24693 leader_election.cc:288] T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:52.152899 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e92f377a79354a6ebb4eec67303c13e3" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:03:52.153745 24473 raft_consensus.cc:2379] T e92f377a79354a6ebb4eec67303c13e3 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 1: Already voted for candidate eeab93abfdb141a895002ec9f312296c in this term.
I20260925 03:03:52.138059 24548 raft_consensus.cc:2454] T f22ddd7d1c444dc58f45a1fb0b3e02bc P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 1.
I20260925 03:03:52.156563 24711 raft_consensus.cc:504] T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:52.163185 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e92f377a79354a6ebb4eec67303c13e3" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:03:52.169510 24511 leader_election.cc:302] T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:03:52.171041 24725 raft_consensus.cc:3037] T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:52.173707 24663 tablet_bootstrap.cc:492] T 0e54ca1757ee4e42956a9cebbeca4f62 P a04a37fc4aa7460cbfdbeeb4e451e1f5: No bootstrap required, opened a new log
I20260925 03:03:52.174360 24663 ts_tablet_manager.cc:1397] T 0e54ca1757ee4e42956a9cebbeca4f62 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent bootstrapping tablet: real 0.173s user 0.008s sys 0.006s
I20260925 03:03:52.181483 24711 leader_election.cc:288] T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:52.184267 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1e85248d3b04b979dd080caeeb02fdb" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:03:52.184957 24548 raft_consensus.cc:3037] T b1e85248d3b04b979dd080caeeb02fdb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:52.185027 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1e85248d3b04b979dd080caeeb02fdb" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:03:52.185662 24621 raft_consensus.cc:3037] T b1e85248d3b04b979dd080caeeb02fdb P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:52.186081 24370 catalog_manager.cc:5495] T 7993f9c3e2a44f408bbf5d1e395e2306 P a04a37fc4aa7460cbfdbeeb4e451e1f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2). New cstate: current_term: 1 leader_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: UNKNOWN } } }
I20260925 03:03:52.188776 24370 catalog_manager.cc:5495] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2). New cstate: current_term: 1 leader_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: UNKNOWN } } }
I20260925 03:03:52.192889 24548 raft_consensus.cc:2454] T b1e85248d3b04b979dd080caeeb02fdb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 1.
I20260925 03:03:52.194355 24663 raft_consensus.cc:348] T 0e54ca1757ee4e42956a9cebbeca4f62 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:52.195422 24663 raft_consensus.cc:374] T 0e54ca1757ee4e42956a9cebbeca4f62 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:52.195735 24663 raft_consensus.cc:729] T 0e54ca1757ee4e42956a9cebbeca4f62 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Initialized, Role: FOLLOWER
I20260925 03:03:52.195600 24371 catalog_manager.cc:5495] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c reported cstate change: term changed from 0 to 1, leader changed from <none> to eeab93abfdb141a895002ec9f312296c (127.22.193.3). New cstate: current_term: 1 leader_uuid: "eeab93abfdb141a895002ec9f312296c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: HEALTHY } } }
I20260925 03:03:52.196604 24663 consensus_queue.cc:260] T 0e54ca1757ee4e42956a9cebbeca4f62 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:52.194768 24725 raft_consensus.cc:2726] T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 03:03:52.199681 24437 leader_election.cc:302] T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:03:52.200655 24621 raft_consensus.cc:2454] T b1e85248d3b04b979dd080caeeb02fdb P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 1.
I20260925 03:03:52.201144 24663 ts_tablet_manager.cc:1428] T 0e54ca1757ee4e42956a9cebbeca4f62 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent starting tablet: real 0.026s user 0.007s sys 0.000s
I20260925 03:03:52.201126 24711 raft_consensus.cc:2781] T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:03:52.202193 24663 tablet_bootstrap.cc:492] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5: Bootstrap starting.
I20260925 03:03:52.201984 24711 raft_consensus.cc:686] T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 LEADER]: Becoming Leader. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:03:52.202975 24711 consensus_queue.cc:237] T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:52.208461 24664 tablet_bootstrap.cc:492] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5: No bootstrap required, opened a new log
I20260925 03:03:52.209146 24664 ts_tablet_manager.cc:1397] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5: Time spent bootstrapping tablet: real 0.201s user 0.009s sys 0.012s
I20260925 03:03:52.210796 24662 tablet_bootstrap.cc:492] T 293ef69caf514ac98510a68495f54eda P eeab93abfdb141a895002ec9f312296c: No bootstrap required, opened a new log
I20260925 03:03:52.211315 24662 ts_tablet_manager.cc:1397] T 293ef69caf514ac98510a68495f54eda P eeab93abfdb141a895002ec9f312296c: Time spent bootstrapping tablet: real 0.130s user 0.007s sys 0.004s
I20260925 03:03:52.213927 24662 raft_consensus.cc:348] T 293ef69caf514ac98510a68495f54eda P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:52.214668 24662 raft_consensus.cc:374] T 293ef69caf514ac98510a68495f54eda P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:52.214916 24662 raft_consensus.cc:729] T 293ef69caf514ac98510a68495f54eda P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Initialized, Role: FOLLOWER
I20260925 03:03:52.215606 24662 consensus_queue.cc:260] T 293ef69caf514ac98510a68495f54eda P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:52.217586 24662 ts_tablet_manager.cc:1428] T 293ef69caf514ac98510a68495f54eda P eeab93abfdb141a895002ec9f312296c: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260925 03:03:52.218418 24662 tablet_bootstrap.cc:492] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c: Bootstrap starting.
I20260925 03:03:52.219359 24664 raft_consensus.cc:348] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:52.220041 24664 raft_consensus.cc:374] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:52.220295 24664 raft_consensus.cc:729] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Initialized, Role: FOLLOWER
I20260925 03:03:52.220921 24664 consensus_queue.cc:260] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:52.222944 24664 ts_tablet_manager.cc:1428] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5: Time spent starting tablet: real 0.013s user 0.004s sys 0.000s
I20260925 03:03:52.223779 24664 tablet_bootstrap.cc:492] T 06f4a7eaf6534e4986c9fd35911038e4 P 869b0a8575b54d08ae2817fad5dd42d5: Bootstrap starting.
I20260925 03:03:52.224388 24662 tablet_bootstrap.cc:654] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:52.228751 24663 tablet_bootstrap.cc:654] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:52.230711 24370 catalog_manager.cc:5495] T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1). New cstate: current_term: 1 leader_uuid: "869b0a8575b54d08ae2817fad5dd42d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: UNKNOWN } } }
I20260925 03:03:52.232165 24664 tablet_bootstrap.cc:654] T 06f4a7eaf6534e4986c9fd35911038e4 P 869b0a8575b54d08ae2817fad5dd42d5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:52.233302 24717 raft_consensus.cc:482] T 71f461a6e2ff434993b78a2d077f0abd P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:52.233755 24717 raft_consensus.cc:504] T 71f461a6e2ff434993b78a2d077f0abd P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:52.235857 24717 leader_election.cc:288] T 71f461a6e2ff434993b78a2d077f0abd P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:03:52.236965 24711 consensus_queue.cc:1035] T edcf6e6a422d4d198d4fbb0f7d3d845d P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260925 03:03:52.237668 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71f461a6e2ff434993b78a2d077f0abd" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
W20260925 03:03:52.241132 24588 leader_election.cc:341] T 71f461a6e2ff434993b78a2d077f0abd P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:52.242470 24725 raft_consensus.cc:482] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:52.242784 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71f461a6e2ff434993b78a2d077f0abd" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:03:52.242937 24725 raft_consensus.cc:504] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:52.243871 24663 tablet_bootstrap.cc:492] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5: No bootstrap required, opened a new log
I20260925 03:03:52.244354 24663 ts_tablet_manager.cc:1397] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent bootstrapping tablet: real 0.042s user 0.011s sys 0.008s
W20260925 03:03:52.244485 24586 leader_election.cc:341] T 71f461a6e2ff434993b78a2d077f0abd P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:52.245152 24586 leader_election.cc:302] T 71f461a6e2ff434993b78a2d077f0abd P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:03:52.246377 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e725b82985746b39c0cc8f671c95353" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:03:52.246788 24717 raft_consensus.cc:2726] T 71f461a6e2ff434993b78a2d077f0abd P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 03:03:52.247054 24472 raft_consensus.cc:2454] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 0.
I20260925 03:03:52.248677 24514 leader_election.cc:302] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:03:52.249594 24693 raft_consensus.cc:2781] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 03:03:52.249948 24693 raft_consensus.cc:482] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:03:52.250269 24693 raft_consensus.cc:3037] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:52.251111 24663 raft_consensus.cc:348] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:52.251924 24663 raft_consensus.cc:374] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:52.252208 24663 raft_consensus.cc:729] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Initialized, Role: FOLLOWER
I20260925 03:03:52.253149 24717 raft_consensus.cc:482] T 293ef69caf514ac98510a68495f54eda P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:52.253407 24725 leader_election.cc:288] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:52.253728 24717 raft_consensus.cc:504] T 293ef69caf514ac98510a68495f54eda P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:52.255682 24717 leader_election.cc:288] T 293ef69caf514ac98510a68495f54eda P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:03:52.256611 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e725b82985746b39c0cc8f671c95353" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:03:52.257262 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "293ef69caf514ac98510a68495f54eda" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
W20260925 03:03:52.258440 24511 leader_election.cc:341] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:52.259411 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "293ef69caf514ac98510a68495f54eda" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:03:52.258984 24663 consensus_queue.cc:260] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:52.261938 24663 ts_tablet_manager.cc:1428] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent starting tablet: real 0.017s user 0.000s sys 0.006s
I20260925 03:03:52.262885 24663 tablet_bootstrap.cc:492] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Bootstrap starting.
I20260925 03:03:52.264173 24711 consensus_queue.cc:1035] T edcf6e6a422d4d198d4fbb0f7d3d845d P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 03:03:52.267027 24472 raft_consensus.cc:2379] T 293ef69caf514ac98510a68495f54eda P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c in current term 1: Already voted for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in this term.
I20260925 03:03:52.267027 24693 raft_consensus.cc:504] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:52.268374 24588 leader_election.cc:302] T 293ef69caf514ac98510a68495f54eda P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:03:52.269194 24716 raft_consensus.cc:3037] T 293ef69caf514ac98510a68495f54eda P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:52.271386 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e725b82985746b39c0cc8f671c95353" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
W20260925 03:03:52.272910 24511 leader_election.cc:341] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:52.273442 24663 tablet_bootstrap.cc:654] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:52.274000 24693 leader_election.cc:288] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:52.277482 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e725b82985746b39c0cc8f671c95353" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:03:52.278103 24472 raft_consensus.cc:3037] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:52.278834 24716 raft_consensus.cc:2726] T 293ef69caf514ac98510a68495f54eda P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 03:03:52.293802 24472 raft_consensus.cc:2454] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 1.
I20260925 03:03:52.297365 24514 leader_election.cc:302] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:03:52.298336 24693 raft_consensus.cc:2781] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:03:52.298849 24693 raft_consensus.cc:686] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 LEADER]: Becoming Leader. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:03:52.299716 24664 tablet_bootstrap.cc:492] T 06f4a7eaf6534e4986c9fd35911038e4 P 869b0a8575b54d08ae2817fad5dd42d5: No bootstrap required, opened a new log
I20260925 03:03:52.300300 24664 ts_tablet_manager.cc:1397] T 06f4a7eaf6534e4986c9fd35911038e4 P 869b0a8575b54d08ae2817fad5dd42d5: Time spent bootstrapping tablet: real 0.077s user 0.006s sys 0.007s
I20260925 03:03:52.299901 24693 consensus_queue.cc:237] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:52.308095 24716 raft_consensus.cc:482] T 48ca6ccd98194cf2af7b5278d15bf4ea P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:52.303200 24664 raft_consensus.cc:348] T 06f4a7eaf6534e4986c9fd35911038e4 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:52.309119 24664 raft_consensus.cc:374] T 06f4a7eaf6534e4986c9fd35911038e4 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:52.309623 24664 raft_consensus.cc:729] T 06f4a7eaf6534e4986c9fd35911038e4 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Initialized, Role: FOLLOWER
I20260925 03:03:52.309064 24716 raft_consensus.cc:504] T 48ca6ccd98194cf2af7b5278d15bf4ea P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:52.310525 24664 consensus_queue.cc:260] T 06f4a7eaf6534e4986c9fd35911038e4 P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:52.312404 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48ca6ccd98194cf2af7b5278d15bf4ea" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:03:52.313339 24664 ts_tablet_manager.cc:1428] T 06f4a7eaf6534e4986c9fd35911038e4 P 869b0a8575b54d08ae2817fad5dd42d5: Time spent starting tablet: real 0.013s user 0.006s sys 0.001s
I20260925 03:03:52.314502 24725 raft_consensus.cc:482] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:52.315006 24725 raft_consensus.cc:504] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:52.315421 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48ca6ccd98194cf2af7b5278d15bf4ea" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:03:52.316579 24547 raft_consensus.cc:2379] T 48ca6ccd98194cf2af7b5278d15bf4ea P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c in current term 1: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:03:52.317390 24725 leader_election.cc:288] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:52.318461 24586 leader_election.cc:302] T 48ca6ccd98194cf2af7b5278d15bf4ea P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:03:52.319485 24717 raft_consensus.cc:3037] T 48ca6ccd98194cf2af7b5278d15bf4ea P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:52.325826 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ac285175464fbda67211d9574db415" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:03:52.332373 24664 tablet_bootstrap.cc:492] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5: Bootstrap starting.
W20260925 03:03:52.332615 24511 leader_election.cc:341] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:52.334941 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ac285175464fbda67211d9574db415" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:03:52.336081 24663 tablet_bootstrap.cc:492] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5: No bootstrap required, opened a new log
I20260925 03:03:52.336531 24663 ts_tablet_manager.cc:1397] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent bootstrapping tablet: real 0.074s user 0.016s sys 0.035s
I20260925 03:03:52.337384 24716 leader_election.cc:288] T 48ca6ccd98194cf2af7b5278d15bf4ea P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:03:52.339212 24663 raft_consensus.cc:348] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:52.340240 24663 raft_consensus.cc:374] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:52.340512 24663 raft_consensus.cc:729] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Initialized, Role: FOLLOWER
W20260925 03:03:52.336428 24514 leader_election.cc:341] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:52.341156 24663 consensus_queue.cc:260] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:52.341804 24514 leader_election.cc:302] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:03:52.342901 24662 tablet_bootstrap.cc:492] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c: No bootstrap required, opened a new log
I20260925 03:03:52.343043 24725 raft_consensus.cc:2726] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 03:03:52.343601 24662 ts_tablet_manager.cc:1397] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c: Time spent bootstrapping tablet: real 0.125s user 0.033s sys 0.034s
I20260925 03:03:52.346084 24664 tablet_bootstrap.cc:654] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:52.346155 24662 raft_consensus.cc:348] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:52.346997 24662 raft_consensus.cc:374] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:52.347263 24662 raft_consensus.cc:729] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Initialized, Role: FOLLOWER
I20260925 03:03:52.348054 24662 consensus_queue.cc:260] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:52.349944 24663 ts_tablet_manager.cc:1428] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20260925 03:03:52.350258 24662 ts_tablet_manager.cc:1428] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 03:03:52.351107 24662 tablet_bootstrap.cc:492] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c: Bootstrap starting.
I20260925 03:03:52.351121 24663 tablet_bootstrap.cc:492] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Bootstrap starting.
I20260925 03:03:52.364557 24725 raft_consensus.cc:482] T 0e54ca1757ee4e42956a9cebbeca4f62 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:52.363620 24369 catalog_manager.cc:5495] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2). New cstate: current_term: 1 leader_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: UNKNOWN } } }
I20260925 03:03:52.365128 24725 raft_consensus.cc:504] T 0e54ca1757ee4e42956a9cebbeca4f62 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:52.367411 24725 leader_election.cc:288] T 0e54ca1757ee4e42956a9cebbeca4f62 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:52.364815 24716 raft_consensus.cc:482] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:52.371232 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e54ca1757ee4e42956a9cebbeca4f62" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:03:52.372298 24717 raft_consensus.cc:2726] T 48ca6ccd98194cf2af7b5278d15bf4ea P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 03:03:52.371877 24716 raft_consensus.cc:504] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:03:52.374671 24511 leader_election.cc:341] T 0e54ca1757ee4e42956a9cebbeca4f62 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:52.374154 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e54ca1757ee4e42956a9cebbeca4f62" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
W20260925 03:03:52.375958 24514 leader_election.cc:341] T 0e54ca1757ee4e42956a9cebbeca4f62 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:52.376423 24514 leader_election.cc:302] T 0e54ca1757ee4e42956a9cebbeca4f62 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:03:52.377288 24725 raft_consensus.cc:2726] T 0e54ca1757ee4e42956a9cebbeca4f62 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 03:03:52.378511 24663 tablet_bootstrap.cc:654] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:52.382876 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e725b82985746b39c0cc8f671c95353" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:03:52.384112 24716 leader_election.cc:288] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:03:52.384514 24472 raft_consensus.cc:2379] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c in current term 1: Already voted for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in this term.
I20260925 03:03:52.386461 24664 tablet_bootstrap.cc:492] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5: No bootstrap required, opened a new log
I20260925 03:03:52.386588 24662 tablet_bootstrap.cc:654] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:52.386904 24664 ts_tablet_manager.cc:1397] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5: Time spent bootstrapping tablet: real 0.055s user 0.017s sys 0.015s
I20260925 03:03:52.387603 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e725b82985746b39c0cc8f671c95353" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:03:52.388890 24663 tablet_bootstrap.cc:492] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5: No bootstrap required, opened a new log
I20260925 03:03:52.389348 24663 ts_tablet_manager.cc:1397] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent bootstrapping tablet: real 0.038s user 0.014s sys 0.007s
I20260925 03:03:52.391407 24586 leader_election.cc:302] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:03:52.391521 24664 raft_consensus.cc:348] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:52.392269 24664 raft_consensus.cc:374] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:52.392544 24664 raft_consensus.cc:729] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Initialized, Role: FOLLOWER
I20260925 03:03:52.392840 24716 raft_consensus.cc:3037] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:52.397400 24664 consensus_queue.cc:260] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:03:52.398026 24418 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260925 03:03:52.399520 24664 ts_tablet_manager.cc:1428] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5: Time spent starting tablet: real 0.012s user 0.002s sys 0.003s
I20260925 03:03:52.403669 24664 tablet_bootstrap.cc:492] T c85f5d1d3f004cf5817bb594fd8e6469 P 869b0a8575b54d08ae2817fad5dd42d5: Bootstrap starting.
I20260925 03:03:52.404933 24717 consensus_queue.cc:1035] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260925 03:03:52.409904 24664 tablet_bootstrap.cc:654] T c85f5d1d3f004cf5817bb594fd8e6469 P 869b0a8575b54d08ae2817fad5dd42d5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:52.409729 24663 raft_consensus.cc:348] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:52.410471 24663 raft_consensus.cc:374] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:52.410759 24663 raft_consensus.cc:729] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Initialized, Role: FOLLOWER
I20260925 03:03:52.411530 24663 consensus_queue.cc:260] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:52.414876 24663 ts_tablet_manager.cc:1428] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent starting tablet: real 0.025s user 0.006s sys 0.000s
I20260925 03:03:52.417587 24663 tablet_bootstrap.cc:492] T c85f5d1d3f004cf5817bb594fd8e6469 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Bootstrap starting.
I20260925 03:03:52.418646 24716 raft_consensus.cc:2726] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 03:03:52.427966 24663 tablet_bootstrap.cc:654] T c85f5d1d3f004cf5817bb594fd8e6469 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:52.428762 24664 tablet_bootstrap.cc:492] T c85f5d1d3f004cf5817bb594fd8e6469 P 869b0a8575b54d08ae2817fad5dd42d5: No bootstrap required, opened a new log
I20260925 03:03:52.429277 24664 ts_tablet_manager.cc:1397] T c85f5d1d3f004cf5817bb594fd8e6469 P 869b0a8575b54d08ae2817fad5dd42d5: Time spent bootstrapping tablet: real 0.026s user 0.019s sys 0.002s
I20260925 03:03:52.431855 24664 raft_consensus.cc:348] T c85f5d1d3f004cf5817bb594fd8e6469 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:52.432585 24664 raft_consensus.cc:374] T c85f5d1d3f004cf5817bb594fd8e6469 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:52.432868 24664 raft_consensus.cc:729] T c85f5d1d3f004cf5817bb594fd8e6469 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Initialized, Role: FOLLOWER
I20260925 03:03:52.433020 24725 raft_consensus.cc:482] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:52.433470 24725 raft_consensus.cc:504] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:52.433590 24664 consensus_queue.cc:260] T c85f5d1d3f004cf5817bb594fd8e6469 P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:52.435510 24725 leader_election.cc:288] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:52.436947 24664 ts_tablet_manager.cc:1428] T c85f5d1d3f004cf5817bb594fd8e6469 P 869b0a8575b54d08ae2817fad5dd42d5: Time spent starting tablet: real 0.007s user 0.005s sys 0.003s
I20260925 03:03:52.437155 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06f4a7eaf6534e4986c9fd35911038e4" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:03:52.437752 24472 raft_consensus.cc:2454] T 06f4a7eaf6534e4986c9fd35911038e4 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 0.
I20260925 03:03:52.438108 24664 tablet_bootstrap.cc:492] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5: Bootstrap starting.
I20260925 03:03:52.439517 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06f4a7eaf6534e4986c9fd35911038e4" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:03:52.441942 24514 leader_election.cc:302] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:03:52.442926 24693 raft_consensus.cc:2781] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260925 03:03:52.443081 24511 leader_election.cc:341] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:52.443289 24693 raft_consensus.cc:482] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:03:52.443603 24693 raft_consensus.cc:3037] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:52.447188 24717 consensus_queue.cc:1035] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 03:03:52.451067 24693 raft_consensus.cc:504] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:52.453966 24723 consensus_queue.cc:1035] T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260925 03:03:52.455099 24664 tablet_bootstrap.cc:654] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:52.454795 24670 consensus_queue.cc:1035] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 03:03:52.456828 24693 leader_election.cc:288] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:52.462378 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06f4a7eaf6534e4986c9fd35911038e4" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:03:52.463021 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06f4a7eaf6534e4986c9fd35911038e4" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:03:52.463652 24472 raft_consensus.cc:3037] T 06f4a7eaf6534e4986c9fd35911038e4 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:52.468021 24662 tablet_bootstrap.cc:492] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c: No bootstrap required, opened a new log
I20260925 03:03:52.468420 24662 ts_tablet_manager.cc:1397] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c: Time spent bootstrapping tablet: real 0.117s user 0.004s sys 0.018s
W20260925 03:03:52.467717 24511 leader_election.cc:341] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:52.482604 24472 raft_consensus.cc:2454] T 06f4a7eaf6534e4986c9fd35911038e4 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 1.
I20260925 03:03:52.483949 24514 leader_election.cc:302] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:03:52.486023 24663 tablet_bootstrap.cc:492] T c85f5d1d3f004cf5817bb594fd8e6469 P a04a37fc4aa7460cbfdbeeb4e451e1f5: No bootstrap required, opened a new log
I20260925 03:03:52.486246 24693 raft_consensus.cc:2781] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:03:52.486718 24693 raft_consensus.cc:686] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 LEADER]: Becoming Leader. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:03:52.486918 24725 raft_consensus.cc:482] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:52.487462 24725 raft_consensus.cc:504] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:52.487600 24693 consensus_queue.cc:237] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:52.494550 24719 consensus_queue.cc:1035] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260925 03:03:52.567350 24719 consensus_queue.cc:1035] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 03:03:52.574301 24671 consensus_queue.cc:1035] T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 03:03:52.575659 24663 ts_tablet_manager.cc:1397] T c85f5d1d3f004cf5817bb594fd8e6469 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent bootstrapping tablet: real 0.158s user 0.018s sys 0.032s
I20260925 03:03:52.574307 24662 raft_consensus.cc:348] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:52.576107 24662 raft_consensus.cc:374] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:52.576308 24662 raft_consensus.cc:729] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Initialized, Role: FOLLOWER
I20260925 03:03:52.576766 24662 consensus_queue.cc:260] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:52.594404 24662 ts_tablet_manager.cc:1428] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c: Time spent starting tablet: real 0.126s user 0.008s sys 0.000s
I20260925 03:03:52.595789 24369 catalog_manager.cc:5495] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2). New cstate: current_term: 1 leader_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: UNKNOWN } } }
I20260925 03:03:52.594183 24671 consensus_queue.cc:1035] T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260925 03:03:52.598377 24667 consensus_queue.cc:1035] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 03:03:52.581069 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71f461a6e2ff434993b78a2d077f0abd" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:03:52.602522 24662 tablet_bootstrap.cc:492] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c: Bootstrap starting.
I20260925 03:03:52.600400 24725 leader_election.cc:288] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
W20260925 03:03:52.603060 24514 leader_election.cc:341] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:52.600719 24671 raft_consensus.cc:482] T c85f5d1d3f004cf5817bb594fd8e6469 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:52.616799 24664 tablet_bootstrap.cc:492] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5: No bootstrap required, opened a new log
I20260925 03:03:52.617632 24664 ts_tablet_manager.cc:1397] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5: Time spent bootstrapping tablet: real 0.180s user 0.010s sys 0.009s
I20260925 03:03:52.612396 24671 raft_consensus.cc:504] T c85f5d1d3f004cf5817bb594fd8e6469 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:52.624766 24664 raft_consensus.cc:348] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:52.625530 24664 raft_consensus.cc:374] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:52.625728 24664 raft_consensus.cc:729] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Initialized, Role: FOLLOWER
I20260925 03:03:52.626394 24664 consensus_queue.cc:260] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:52.621073 24723 raft_consensus.cc:482] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:52.634004 24662 tablet_bootstrap.cc:654] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:52.635815 24664 ts_tablet_manager.cc:1428] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5: Time spent starting tablet: real 0.018s user 0.005s sys 0.000s
I20260925 03:03:52.636909 24664 tablet_bootstrap.cc:492] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5: Bootstrap starting.
I20260925 03:03:52.638463 24671 leader_election.cc:288] T c85f5d1d3f004cf5817bb594fd8e6469 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:52.637934 24723 raft_consensus.cc:504] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:52.644299 24723 leader_election.cc:288] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:52.661926 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c85f5d1d3f004cf5817bb594fd8e6469" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:03:52.658871 24663 raft_consensus.cc:348] T c85f5d1d3f004cf5817bb594fd8e6469 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:52.663643 24663 raft_consensus.cc:374] T c85f5d1d3f004cf5817bb594fd8e6469 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:52.663976 24663 raft_consensus.cc:729] T c85f5d1d3f004cf5817bb594fd8e6469 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Initialized, Role: FOLLOWER
W20260925 03:03:52.665625 24436 leader_election.cc:341] T c85f5d1d3f004cf5817bb594fd8e6469 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:52.671594 24664 tablet_bootstrap.cc:654] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:52.668795 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
W20260925 03:03:52.673328 24514 leader_election.cc:341] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:52.673615 24670 consensus_queue.cc:1035] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 03:03:52.675576 24663 consensus_queue.cc:260] T c85f5d1d3f004cf5817bb594fd8e6469 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:52.677459 24663 ts_tablet_manager.cc:1428] T c85f5d1d3f004cf5817bb594fd8e6469 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent starting tablet: real 0.101s user 0.007s sys 0.000s
W20260925 03:03:52.684434 24511 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 99685.6us
I20260925 03:03:52.683005 24671 consensus_queue.cc:1035] T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260925 03:03:52.735036 24671 raft_consensus.cc:482] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:52.744378 24671 raft_consensus.cc:504] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:52.747370 24663 tablet_bootstrap.cc:492] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Bootstrap starting.
I20260925 03:03:52.747808 24671 leader_election.cc:288] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:52.773818 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71f461a6e2ff434993b78a2d077f0abd" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:03:52.774613 24620 raft_consensus.cc:2454] T 71f461a6e2ff434993b78a2d077f0abd P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 0.
I20260925 03:03:52.778726 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c85f5d1d3f004cf5817bb594fd8e6469" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:03:52.779527 24548 raft_consensus.cc:2454] T c85f5d1d3f004cf5817bb594fd8e6469 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 0.
I20260925 03:03:52.780249 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:03:52.780838 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60d45b7d467d4e1ab92367f6d7d52b15" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:03:52.783901 24725 consensus_queue.cc:1035] T 7993f9c3e2a44f408bbf5d1e395e2306 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260925 03:03:52.788075 24719 raft_consensus.cc:482] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:52.788563 24719 raft_consensus.cc:504] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:52.791457 24719 leader_election.cc:288] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:03:52.783905 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60d45b7d467d4e1ab92367f6d7d52b15" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:03:52.819612 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e54ca1757ee4e42956a9cebbeca4f62" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:03:52.825047 24511 leader_election.cc:302] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5
W20260925 03:03:52.814411 24436 leader_election.cc:341] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:52.815495 24725 raft_consensus.cc:482] T c85f5d1d3f004cf5817bb594fd8e6469 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:52.827488 24671 raft_consensus.cc:482] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:52.832329 24725 raft_consensus.cc:504] T c85f5d1d3f004cf5817bb594fd8e6469 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:52.832809 24671 raft_consensus.cc:504] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:52.834344 24725 leader_election.cc:288] T c85f5d1d3f004cf5817bb594fd8e6469 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:52.834995 24548 raft_consensus.cc:2454] T 0e54ca1757ee4e42956a9cebbeca4f62 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 0.
I20260925 03:03:52.835300 24671 leader_election.cc:288] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:52.836668 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e54ca1757ee4e42956a9cebbeca4f62" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:03:52.828016 24693 consensus_queue.cc:1035] T 7993f9c3e2a44f408bbf5d1e395e2306 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.011s
W20260925 03:03:52.839457 24588 leader_election.cc:341] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:52.840538 24586 leader_election.cc:302] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:03:52.841789 24719 raft_consensus.cc:2781] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 03:03:52.842216 24719 raft_consensus.cc:482] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:03:52.842571 24719 raft_consensus.cc:3037] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:52.847908 24763 raft_consensus.cc:2781] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260925 03:03:52.848003 24511 leader_election.cc:341] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:52.848438 24763 raft_consensus.cc:482] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:03:52.848567 24511 leader_election.cc:302] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:03:52.848861 24763 raft_consensus.cc:3037] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:52.854197 24765 raft_consensus.cc:2726] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 03:03:52.827086 24437 leader_election.cc:302] T c85f5d1d3f004cf5817bb594fd8e6469 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:03:52.863881 24667 consensus_queue.cc:1035] T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 03:03:52.867002 24708 raft_consensus.cc:2781] T c85f5d1d3f004cf5817bb594fd8e6469 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 03:03:52.866837 24618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c85f5d1d3f004cf5817bb594fd8e6469" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:03:52.868769 24708 raft_consensus.cc:482] T c85f5d1d3f004cf5817bb594fd8e6469 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:03:52.869393 24708 raft_consensus.cc:3037] T c85f5d1d3f004cf5817bb594fd8e6469 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:52.879415 24618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b5df303cd6a4d59aca5c18bbbff24ed" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:03:52.884362 24663 tablet_bootstrap.cc:654] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Neither blocks nor log segments found. Creating new log.
W20260925 03:03:52.871544 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 54197.6us
W20260925 03:03:52.886245 24437 leader_election.cc:341] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:52.886958 24437 leader_election.cc:302] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:03:52.888206 24671 raft_consensus.cc:2726] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 03:03:52.888916 24723 consensus_queue.cc:1035] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 03:03:52.890632 24511 leader_election.cc:341] T c85f5d1d3f004cf5817bb594fd8e6469 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:52.894423 24763 raft_consensus.cc:504] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:03:52.895445 24436 leader_election.cc:341] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:52.896836 24771 consensus_queue.cc:1035] T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260925 03:03:52.892064 24719 raft_consensus.cc:504] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:52.899771 24662 tablet_bootstrap.cc:492] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c: No bootstrap required, opened a new log
I20260925 03:03:52.900213 24662 ts_tablet_manager.cc:1397] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c: Time spent bootstrapping tablet: real 0.298s user 0.020s sys 0.012s
I20260925 03:03:52.906876 24708 raft_consensus.cc:504] T c85f5d1d3f004cf5817bb594fd8e6469 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:52.910758 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b5df303cd6a4d59aca5c18bbbff24ed" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:03:52.911008 24708 leader_election.cc:288] T c85f5d1d3f004cf5817bb594fd8e6469 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:52.911091 24719 leader_election.cc:288] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 1 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:03:52.911520 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e54ca1757ee4e42956a9cebbeca4f62" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:03:52.913460 24763 leader_election.cc:288] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:52.913546 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e54ca1757ee4e42956a9cebbeca4f62" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:03:52.914161 24547 raft_consensus.cc:3037] T 0e54ca1757ee4e42956a9cebbeca4f62 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:52.915907 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c85f5d1d3f004cf5817bb594fd8e6469" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:03:52.916268 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c85f5d1d3f004cf5817bb594fd8e6469" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:03:52.916734 24473 raft_consensus.cc:2379] T c85f5d1d3f004cf5817bb594fd8e6469 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 1: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
W20260925 03:03:52.917574 24436 leader_election.cc:341] T c85f5d1d3f004cf5817bb594fd8e6469 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105): Illegal state: must be running to vote when last-logged opid is not known
W20260925 03:03:52.918545 24588 leader_election.cc:341] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:52.926970 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71f461a6e2ff434993b78a2d077f0abd" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:03:52.927604 24620 raft_consensus.cc:3037] T 71f461a6e2ff434993b78a2d077f0abd P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:52.946049 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71f461a6e2ff434993b78a2d077f0abd" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:03:52.970664 24547 raft_consensus.cc:2454] T 0e54ca1757ee4e42956a9cebbeca4f62 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 1.
I20260925 03:03:52.971968 24586 leader_election.cc:302] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:03:52.973091 24719 raft_consensus.cc:2781] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:03:52.973567 24719 raft_consensus.cc:686] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [term 1 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:03:52.974336 24719 consensus_queue.cc:237] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:52.979089 24620 raft_consensus.cc:2454] T 71f461a6e2ff434993b78a2d077f0abd P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 1.
I20260925 03:03:52.983145 24662 raft_consensus.cc:348] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:52.991830 24514 leader_election.cc:302] T c85f5d1d3f004cf5817bb594fd8e6469 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:03:52.992944 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c85f5d1d3f004cf5817bb594fd8e6469" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:03:52.995666 24763 consensus_queue.cc:1035] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260925 03:03:53.000192 24662 raft_consensus.cc:374] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:53.000571 24662 raft_consensus.cc:729] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Initialized, Role: FOLLOWER
I20260925 03:03:53.001260 24662 consensus_queue.cc:260] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:53.008258 24511 leader_election.cc:302] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:03:53.011155 24769 raft_consensus.cc:2781] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:03:53.012590 24769 raft_consensus.cc:686] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 LEADER]: Becoming Leader. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:03:53.004570 24546 raft_consensus.cc:3037] T c85f5d1d3f004cf5817bb594fd8e6469 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:53.018317 24667 consensus_queue.cc:1035] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 03:03:53.019003 24514 leader_election.cc:341] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:53.020522 24770 consensus_queue.cc:1035] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260925 03:03:53.030179 24662 ts_tablet_manager.cc:1428] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c: Time spent starting tablet: real 0.130s user 0.011s sys 0.003s
I20260925 03:03:53.030171 24664 tablet_bootstrap.cc:492] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5: No bootstrap required, opened a new log
I20260925 03:03:53.030242 24663 tablet_bootstrap.cc:492] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5: No bootstrap required, opened a new log
I20260925 03:03:53.030949 24664 ts_tablet_manager.cc:1397] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5: Time spent bootstrapping tablet: real 0.394s user 0.036s sys 0.046s
I20260925 03:03:53.031013 24663 ts_tablet_manager.cc:1397] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent bootstrapping tablet: real 0.284s user 0.015s sys 0.012s
I20260925 03:03:53.031410 24662 tablet_bootstrap.cc:492] T 0b5df303cd6a4d59aca5c18bbbff24ed P eeab93abfdb141a895002ec9f312296c: Bootstrap starting.
I20260925 03:03:53.013370 24769 consensus_queue.cc:237] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:03:53.016155 24437 leader_election.cc:341] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:53.051986 24717 raft_consensus.cc:482] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:53.052651 24717 raft_consensus.cc:504] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:53.054628 24663 raft_consensus.cc:348] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:53.055431 24663 raft_consensus.cc:374] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:53.055693 24663 raft_consensus.cc:729] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Initialized, Role: FOLLOWER
I20260925 03:03:53.056527 24717 leader_election.cc:288] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:03:53.056403 24663 consensus_queue.cc:260] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:53.057258 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
W20260925 03:03:53.058885 24588 leader_election.cc:341] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:53.059274 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:03:53.059815 24547 raft_consensus.cc:2454] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 0.
W20260925 03:03:53.058843 24511 consensus_peers.cc:487] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 -> Peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105): Couldn't send request to peer eeab93abfdb141a895002ec9f312296c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260925 03:03:53.060863 24586 leader_election.cc:302] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:03:53.061218 24664 raft_consensus.cc:348] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:53.061827 24670 raft_consensus.cc:2781] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 03:03:53.062047 24664 raft_consensus.cc:374] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:53.062448 24664 raft_consensus.cc:729] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Initialized, Role: FOLLOWER
I20260925 03:03:53.062628 24670 raft_consensus.cc:482] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:03:53.063653 24663 ts_tablet_manager.cc:1428] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent starting tablet: real 0.032s user 0.010s sys 0.007s
I20260925 03:03:53.063143 24664 consensus_queue.cc:260] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:53.063802 24670 raft_consensus.cc:3037] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:53.064657 24663 tablet_bootstrap.cc:492] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5: Bootstrap starting.
I20260925 03:03:53.065759 24664 ts_tablet_manager.cc:1428] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5: Time spent starting tablet: real 0.034s user 0.006s sys 0.001s
I20260925 03:03:53.067256 24664 tablet_bootstrap.cc:492] T 71f461a6e2ff434993b78a2d077f0abd P 869b0a8575b54d08ae2817fad5dd42d5: Bootstrap starting.
I20260925 03:03:53.055611 24437 leader_election.cc:302] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:03:53.077055 24671 raft_consensus.cc:2726] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260925 03:03:53.079509 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 64694.7us
I20260925 03:03:53.079665 24546 raft_consensus.cc:2454] T c85f5d1d3f004cf5817bb594fd8e6469 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 1.
I20260925 03:03:53.080559 24693 raft_consensus.cc:2726] T c85f5d1d3f004cf5817bb594fd8e6469 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 03:03:53.080657 24670 raft_consensus.cc:504] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:53.084192 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:03:53.085206 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:03:53.085805 24547 raft_consensus.cc:3037] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:53.087345 24662 tablet_bootstrap.cc:654] T 0b5df303cd6a4d59aca5c18bbbff24ed P eeab93abfdb141a895002ec9f312296c: Neither blocks nor log segments found. Creating new log.
W20260925 03:03:53.089253 24588 leader_election.cc:341] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:53.095079 24664 tablet_bootstrap.cc:654] T 71f461a6e2ff434993b78a2d077f0abd P 869b0a8575b54d08ae2817fad5dd42d5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:53.095786 24663 tablet_bootstrap.cc:654] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:53.096964 24670 leader_election.cc:288] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 1 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:03:53.108633 24547 raft_consensus.cc:2454] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 1.
I20260925 03:03:53.112608 24586 leader_election.cc:302] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:03:53.114171 24719 raft_consensus.cc:2781] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:03:53.114667 24719 raft_consensus.cc:686] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [term 1 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:03:53.111344 24368 catalog_manager.cc:5495] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c reported cstate change: term changed from 0 to 1, leader changed from <none> to eeab93abfdb141a895002ec9f312296c (127.22.193.3). New cstate: current_term: 1 leader_uuid: "eeab93abfdb141a895002ec9f312296c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: HEALTHY } } }
I20260925 03:03:53.121914 24719 consensus_queue.cc:237] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:53.123265 24437 leader_election.cc:302] T c85f5d1d3f004cf5817bb594fd8e6469 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:03:53.125092 24771 raft_consensus.cc:2781] T c85f5d1d3f004cf5817bb594fd8e6469 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:03:53.125583 24771 raft_consensus.cc:686] T c85f5d1d3f004cf5817bb594fd8e6469 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 LEADER]: Becoming Leader. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:03:53.126365 24771 consensus_queue.cc:237] T c85f5d1d3f004cf5817bb594fd8e6469 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:53.177271 24664 tablet_bootstrap.cc:492] T 71f461a6e2ff434993b78a2d077f0abd P 869b0a8575b54d08ae2817fad5dd42d5: No bootstrap required, opened a new log
I20260925 03:03:53.177950 24664 ts_tablet_manager.cc:1397] T 71f461a6e2ff434993b78a2d077f0abd P 869b0a8575b54d08ae2817fad5dd42d5: Time spent bootstrapping tablet: real 0.111s user 0.015s sys 0.010s
I20260925 03:03:53.185848 24662 tablet_bootstrap.cc:492] T 0b5df303cd6a4d59aca5c18bbbff24ed P eeab93abfdb141a895002ec9f312296c: No bootstrap required, opened a new log
I20260925 03:03:53.186465 24662 ts_tablet_manager.cc:1397] T 0b5df303cd6a4d59aca5c18bbbff24ed P eeab93abfdb141a895002ec9f312296c: Time spent bootstrapping tablet: real 0.155s user 0.014s sys 0.023s
I20260925 03:03:53.189297 24662 raft_consensus.cc:348] T 0b5df303cd6a4d59aca5c18bbbff24ed P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:53.190110 24662 raft_consensus.cc:374] T 0b5df303cd6a4d59aca5c18bbbff24ed P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:53.190409 24662 raft_consensus.cc:729] T 0b5df303cd6a4d59aca5c18bbbff24ed P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Initialized, Role: FOLLOWER
I20260925 03:03:53.190719 24664 raft_consensus.cc:348] T 71f461a6e2ff434993b78a2d077f0abd P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:53.191035 24662 consensus_queue.cc:260] T 0b5df303cd6a4d59aca5c18bbbff24ed P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:53.191511 24664 raft_consensus.cc:374] T 71f461a6e2ff434993b78a2d077f0abd P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:53.191855 24664 raft_consensus.cc:729] T 71f461a6e2ff434993b78a2d077f0abd P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Initialized, Role: FOLLOWER
I20260925 03:03:53.192556 24664 consensus_queue.cc:260] T 71f461a6e2ff434993b78a2d077f0abd P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:53.196477 24664 ts_tablet_manager.cc:1428] T 71f461a6e2ff434993b78a2d077f0abd P 869b0a8575b54d08ae2817fad5dd42d5: Time spent starting tablet: real 0.018s user 0.007s sys 0.000s
I20260925 03:03:53.197504 24664 tablet_bootstrap.cc:492] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5: Bootstrap starting.
I20260925 03:03:53.202016 24662 ts_tablet_manager.cc:1428] T 0b5df303cd6a4d59aca5c18bbbff24ed P eeab93abfdb141a895002ec9f312296c: Time spent starting tablet: real 0.015s user 0.008s sys 0.000s
I20260925 03:03:53.203516 24369 catalog_manager.cc:5495] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2). New cstate: current_term: 1 leader_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: UNKNOWN } } }
I20260925 03:03:53.208083 24662 tablet_bootstrap.cc:492] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c: Bootstrap starting.
I20260925 03:03:53.217751 24662 tablet_bootstrap.cc:654] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:53.220314 24663 tablet_bootstrap.cc:492] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5: No bootstrap required, opened a new log
I20260925 03:03:53.220916 24663 ts_tablet_manager.cc:1397] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent bootstrapping tablet: real 0.156s user 0.037s sys 0.036s
I20260925 03:03:53.225035 24663 raft_consensus.cc:348] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:53.224325 24370 catalog_manager.cc:5495] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c reported cstate change: term changed from 0 to 1, leader changed from <none> to eeab93abfdb141a895002ec9f312296c (127.22.193.3). New cstate: current_term: 1 leader_uuid: "eeab93abfdb141a895002ec9f312296c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: HEALTHY } } }
I20260925 03:03:53.225900 24663 raft_consensus.cc:374] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:53.226219 24663 raft_consensus.cc:729] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Initialized, Role: FOLLOWER
I20260925 03:03:53.230644 24662 tablet_bootstrap.cc:492] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c: No bootstrap required, opened a new log
I20260925 03:03:53.231109 24662 ts_tablet_manager.cc:1397] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c: Time spent bootstrapping tablet: real 0.023s user 0.015s sys 0.000s
I20260925 03:03:53.231590 24664 tablet_bootstrap.cc:654] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:53.228427 24663 consensus_queue.cc:260] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:53.234090 24662 raft_consensus.cc:348] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:53.239821 24662 raft_consensus.cc:374] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:53.240125 24662 raft_consensus.cc:729] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Initialized, Role: FOLLOWER
I20260925 03:03:53.240927 24663 ts_tablet_manager.cc:1428] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent starting tablet: real 0.020s user 0.009s sys 0.002s
I20260925 03:03:53.240865 24662 consensus_queue.cc:260] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:53.242695 24663 tablet_bootstrap.cc:492] T 0b5df303cd6a4d59aca5c18bbbff24ed P a04a37fc4aa7460cbfdbeeb4e451e1f5: Bootstrap starting.
I20260925 03:03:53.243242 24662 ts_tablet_manager.cc:1428] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c: Time spent starting tablet: real 0.012s user 0.008s sys 0.003s
I20260925 03:03:53.244079 24662 tablet_bootstrap.cc:492] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c: Bootstrap starting.
I20260925 03:03:53.245044 24368 catalog_manager.cc:5495] T c85f5d1d3f004cf5817bb594fd8e6469 P 869b0a8575b54d08ae2817fad5dd42d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1). New cstate: current_term: 1 leader_uuid: "869b0a8575b54d08ae2817fad5dd42d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: UNKNOWN } } }
I20260925 03:03:53.258060 24664 tablet_bootstrap.cc:492] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5: No bootstrap required, opened a new log
I20260925 03:03:53.258548 24664 ts_tablet_manager.cc:1397] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5: Time spent bootstrapping tablet: real 0.061s user 0.015s sys 0.006s
I20260925 03:03:53.260563 24662 tablet_bootstrap.cc:654] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:53.261385 24664 raft_consensus.cc:348] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:53.262106 24664 raft_consensus.cc:374] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:53.262367 24664 raft_consensus.cc:729] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Initialized, Role: FOLLOWER
I20260925 03:03:53.263095 24664 consensus_queue.cc:260] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:53.265532 24664 ts_tablet_manager.cc:1428] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5: Time spent starting tablet: real 0.007s user 0.007s sys 0.001s
I20260925 03:03:53.266124 24663 tablet_bootstrap.cc:654] T 0b5df303cd6a4d59aca5c18bbbff24ed P a04a37fc4aa7460cbfdbeeb4e451e1f5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:53.266961 24664 tablet_bootstrap.cc:492] T b627ecb4a5384ef392aa695e06599c39 P 869b0a8575b54d08ae2817fad5dd42d5: Bootstrap starting.
I20260925 03:03:53.273258 24664 tablet_bootstrap.cc:654] T b627ecb4a5384ef392aa695e06599c39 P 869b0a8575b54d08ae2817fad5dd42d5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:53.280424 24663 tablet_bootstrap.cc:492] T 0b5df303cd6a4d59aca5c18bbbff24ed P a04a37fc4aa7460cbfdbeeb4e451e1f5: No bootstrap required, opened a new log
I20260925 03:03:53.280910 24663 ts_tablet_manager.cc:1397] T 0b5df303cd6a4d59aca5c18bbbff24ed P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent bootstrapping tablet: real 0.039s user 0.010s sys 0.012s
I20260925 03:03:53.283893 24662 tablet_bootstrap.cc:492] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c: No bootstrap required, opened a new log
I20260925 03:03:53.284358 24662 ts_tablet_manager.cc:1397] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c: Time spent bootstrapping tablet: real 0.040s user 0.008s sys 0.012s
I20260925 03:03:53.287659 24663 raft_consensus.cc:348] T 0b5df303cd6a4d59aca5c18bbbff24ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:53.288151 24662 raft_consensus.cc:348] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:53.288408 24663 raft_consensus.cc:374] T 0b5df303cd6a4d59aca5c18bbbff24ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:53.288727 24662 raft_consensus.cc:374] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:53.288834 24664 tablet_bootstrap.cc:492] T b627ecb4a5384ef392aa695e06599c39 P 869b0a8575b54d08ae2817fad5dd42d5: No bootstrap required, opened a new log
I20260925 03:03:53.288779 24663 raft_consensus.cc:729] T 0b5df303cd6a4d59aca5c18bbbff24ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Initialized, Role: FOLLOWER
I20260925 03:03:53.289073 24662 raft_consensus.cc:729] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Initialized, Role: FOLLOWER
I20260925 03:03:53.289316 24664 ts_tablet_manager.cc:1397] T b627ecb4a5384ef392aa695e06599c39 P 869b0a8575b54d08ae2817fad5dd42d5: Time spent bootstrapping tablet: real 0.023s user 0.012s sys 0.000s
I20260925 03:03:53.289656 24663 consensus_queue.cc:260] T 0b5df303cd6a4d59aca5c18bbbff24ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:53.289841 24662 consensus_queue.cc:260] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:53.291970 24663 ts_tablet_manager.cc:1428] T 0b5df303cd6a4d59aca5c18bbbff24ed P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent starting tablet: real 0.011s user 0.008s sys 0.000s
I20260925 03:03:53.291841 24664 raft_consensus.cc:348] T b627ecb4a5384ef392aa695e06599c39 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:53.292623 24664 raft_consensus.cc:374] T b627ecb4a5384ef392aa695e06599c39 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:53.292918 24664 raft_consensus.cc:729] T b627ecb4a5384ef392aa695e06599c39 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Initialized, Role: FOLLOWER
I20260925 03:03:53.293236 24663 tablet_bootstrap.cc:492] T 8ac466793d094325ace224708b2158ed P a04a37fc4aa7460cbfdbeeb4e451e1f5: Bootstrap starting.
I20260925 03:03:53.293462 24664 consensus_queue.cc:260] T b627ecb4a5384ef392aa695e06599c39 P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:53.295898 24664 ts_tablet_manager.cc:1428] T b627ecb4a5384ef392aa695e06599c39 P 869b0a8575b54d08ae2817fad5dd42d5: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260925 03:03:53.296716 24664 tablet_bootstrap.cc:492] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5: Bootstrap starting.
I20260925 03:03:53.297133 24662 ts_tablet_manager.cc:1428] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c: Time spent starting tablet: real 0.013s user 0.008s sys 0.001s
I20260925 03:03:53.300359 24663 tablet_bootstrap.cc:654] T 8ac466793d094325ace224708b2158ed P a04a37fc4aa7460cbfdbeeb4e451e1f5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:53.301534 24662 tablet_bootstrap.cc:492] T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c: Bootstrap starting.
I20260925 03:03:53.306926 24771 raft_consensus.cc:482] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:53.307669 24771 raft_consensus.cc:504] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:53.313429 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ac285175464fbda67211d9574db415" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:03:53.312580 24771 leader_election.cc:288] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:53.314728 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ac285175464fbda67211d9574db415" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
W20260925 03:03:53.315037 24436 leader_election.cc:341] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:53.315352 24547 raft_consensus.cc:2454] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 0.
I20260925 03:03:53.316522 24664 tablet_bootstrap.cc:654] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:53.316458 24437 leader_election.cc:302] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:03:53.317230 24771 raft_consensus.cc:2781] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 03:03:53.317863 24771 raft_consensus.cc:482] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:03:53.318449 24771 raft_consensus.cc:3037] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:53.322055 24662 tablet_bootstrap.cc:654] T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:53.331931 24663 tablet_bootstrap.cc:492] T 8ac466793d094325ace224708b2158ed P a04a37fc4aa7460cbfdbeeb4e451e1f5: No bootstrap required, opened a new log
I20260925 03:03:53.332676 24663 ts_tablet_manager.cc:1397] T 8ac466793d094325ace224708b2158ed P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent bootstrapping tablet: real 0.040s user 0.007s sys 0.010s
I20260925 03:03:53.333230 24771 raft_consensus.cc:504] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:53.335942 24663 raft_consensus.cc:348] T 8ac466793d094325ace224708b2158ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:53.336628 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ac285175464fbda67211d9574db415" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:03:53.337385 24547 raft_consensus.cc:3037] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:53.336997 24663 raft_consensus.cc:374] T 8ac466793d094325ace224708b2158ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:53.337896 24663 raft_consensus.cc:729] T 8ac466793d094325ace224708b2158ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Initialized, Role: FOLLOWER
I20260925 03:03:53.339438 24663 consensus_queue.cc:260] T 8ac466793d094325ace224708b2158ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:53.340268 24671 raft_consensus.cc:482] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:53.341078 24671 raft_consensus.cc:504] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:53.343014 24663 ts_tablet_manager.cc:1428] T 8ac466793d094325ace224708b2158ed P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20260925 03:03:53.343539 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ac285175464fbda67211d9574db415" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:03:53.344184 24663 tablet_bootstrap.cc:492] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Bootstrap starting.
I20260925 03:03:53.344986 24671 leader_election.cc:288] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:53.345475 24664 tablet_bootstrap.cc:492] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5: No bootstrap required, opened a new log
W20260925 03:03:53.345606 24436 leader_election.cc:341] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:53.346047 24664 ts_tablet_manager.cc:1397] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5: Time spent bootstrapping tablet: real 0.050s user 0.020s sys 0.007s
I20260925 03:03:53.346984 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5aec38e66a947c681e6c2a72b05e6ca" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:03:53.347616 24622 raft_consensus.cc:2454] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 0.
I20260925 03:03:53.348733 24664 raft_consensus.cc:348] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:53.348963 24436 leader_election.cc:302] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:03:53.349499 24771 leader_election.cc:288] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:53.349860 24663 tablet_bootstrap.cc:654] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:53.349821 24671 raft_consensus.cc:2781] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 03:03:53.350382 24671 raft_consensus.cc:482] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:03:53.350721 24671 raft_consensus.cc:3037] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:53.351866 24662 tablet_bootstrap.cc:492] T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c: No bootstrap required, opened a new log
I20260925 03:03:53.352403 24662 ts_tablet_manager.cc:1397] T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c: Time spent bootstrapping tablet: real 0.051s user 0.023s sys 0.006s
I20260925 03:03:53.349733 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5aec38e66a947c681e6c2a72b05e6ca" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
W20260925 03:03:53.355178 24437 leader_election.cc:341] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:53.357182 24771 raft_consensus.cc:482] T 71f461a6e2ff434993b78a2d077f0abd P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:53.357684 24662 raft_consensus.cc:348] T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:53.358450 24662 raft_consensus.cc:374] T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:53.358743 24662 raft_consensus.cc:729] T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Initialized, Role: FOLLOWER
I20260925 03:03:53.359032 24547 raft_consensus.cc:2454] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 1.
I20260925 03:03:53.359534 24662 consensus_queue.cc:260] T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:53.360714 24664 raft_consensus.cc:374] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:53.360296 24437 leader_election.cc:302] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:03:53.357656 24771 raft_consensus.cc:504] T 71f461a6e2ff434993b78a2d077f0abd P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:53.361745 24711 raft_consensus.cc:2781] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:03:53.362264 24711 raft_consensus.cc:686] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 LEADER]: Becoming Leader. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:03:53.364503 24711 consensus_queue.cc:237] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:53.365850 24662 ts_tablet_manager.cc:1428] T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c: Time spent starting tablet: real 0.013s user 0.008s sys 0.000s
I20260925 03:03:53.366901 24662 tablet_bootstrap.cc:492] T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c: Bootstrap starting.
I20260925 03:03:53.367555 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71f461a6e2ff434993b78a2d077f0abd" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:03:53.367364 24671 raft_consensus.cc:504] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:53.361001 24664 raft_consensus.cc:729] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Initialized, Role: FOLLOWER
I20260925 03:03:53.373152 24664 consensus_queue.cc:260] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:53.379866 24664 ts_tablet_manager.cc:1428] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5: Time spent starting tablet: real 0.033s user 0.007s sys 0.000s
I20260925 03:03:53.381040 24664 tablet_bootstrap.cc:492] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5: Bootstrap starting.
I20260925 03:03:53.372907 24771 leader_election.cc:288] T 71f461a6e2ff434993b78a2d077f0abd P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:53.383471 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71f461a6e2ff434993b78a2d077f0abd" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:03:53.384747 24622 raft_consensus.cc:2379] T 71f461a6e2ff434993b78a2d077f0abd P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 in current term 1: Already voted for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in this term.
I20260925 03:03:53.385020 24671 leader_election.cc:288] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:53.385720 24436 leader_election.cc:302] T 71f461a6e2ff434993b78a2d077f0abd P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:03:53.386394 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5aec38e66a947c681e6c2a72b05e6ca" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:03:53.386967 24622 raft_consensus.cc:3037] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:53.392871 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5aec38e66a947c681e6c2a72b05e6ca" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
W20260925 03:03:53.396770 24437 leader_election.cc:341] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:53.386998 24671 raft_consensus.cc:3037] T 71f461a6e2ff434993b78a2d077f0abd P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:53.402053 24662 tablet_bootstrap.cc:654] T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:53.404282 24663 tablet_bootstrap.cc:492] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5: No bootstrap required, opened a new log
I20260925 03:03:53.411835 24663 ts_tablet_manager.cc:1397] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent bootstrapping tablet: real 0.068s user 0.015s sys 0.020s
I20260925 03:03:53.416395 24664 tablet_bootstrap.cc:654] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:53.416810 24671 raft_consensus.cc:2726] T 71f461a6e2ff434993b78a2d077f0abd P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 03:03:53.416483 24663 raft_consensus.cc:348] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:53.417373 24663 raft_consensus.cc:374] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:53.417668 24663 raft_consensus.cc:729] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Initialized, Role: FOLLOWER
I20260925 03:03:53.418455 24663 consensus_queue.cc:260] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:53.424088 24663 ts_tablet_manager.cc:1428] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent starting tablet: real 0.012s user 0.008s sys 0.000s
I20260925 03:03:53.425020 24663 tablet_bootstrap.cc:492] T c5aec38e66a947c681e6c2a72b05e6ca P a04a37fc4aa7460cbfdbeeb4e451e1f5: Bootstrap starting.
I20260925 03:03:53.425434 24622 raft_consensus.cc:2454] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 1.
I20260925 03:03:53.427532 24370 catalog_manager.cc:5495] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1). New cstate: current_term: 1 leader_uuid: "869b0a8575b54d08ae2817fad5dd42d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: UNKNOWN } } }
I20260925 03:03:53.430871 24436 leader_election.cc:302] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:03:53.440999 24663 tablet_bootstrap.cc:654] T c5aec38e66a947c681e6c2a72b05e6ca P a04a37fc4aa7460cbfdbeeb4e451e1f5: Neither blocks nor log segments found. Creating new log.
W20260925 03:03:53.456017 24418 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260925 03:03:53.449694 24671 raft_consensus.cc:2781] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:03:53.458021 24671 raft_consensus.cc:686] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 LEADER]: Becoming Leader. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:03:53.459780 24671 consensus_queue.cc:237] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:53.461968 24664 tablet_bootstrap.cc:492] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5: No bootstrap required, opened a new log
I20260925 03:03:53.462445 24664 ts_tablet_manager.cc:1397] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5: Time spent bootstrapping tablet: real 0.082s user 0.012s sys 0.007s
I20260925 03:03:53.465734 24664 raft_consensus.cc:348] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:53.466398 24664 raft_consensus.cc:374] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:53.466665 24664 raft_consensus.cc:729] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Initialized, Role: FOLLOWER
I20260925 03:03:53.467237 24664 consensus_queue.cc:260] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:53.469269 24664 ts_tablet_manager.cc:1428] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260925 03:03:53.470142 24664 tablet_bootstrap.cc:492] T 4e5332a5d329493991dff994e70ea92c P 869b0a8575b54d08ae2817fad5dd42d5: Bootstrap starting.
I20260925 03:03:53.470058 24370 catalog_manager.cc:5495] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1). New cstate: current_term: 1 leader_uuid: "869b0a8575b54d08ae2817fad5dd42d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: UNKNOWN } } }
I20260925 03:03:53.492967 24664 tablet_bootstrap.cc:654] T 4e5332a5d329493991dff994e70ea92c P 869b0a8575b54d08ae2817fad5dd42d5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:53.492933 24769 raft_consensus.cc:482] T 0b5df303cd6a4d59aca5c18bbbff24ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:53.493940 24662 tablet_bootstrap.cc:492] T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c: No bootstrap required, opened a new log
I20260925 03:03:53.493773 24769 raft_consensus.cc:504] T 0b5df303cd6a4d59aca5c18bbbff24ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:53.494429 24663 tablet_bootstrap.cc:492] T c5aec38e66a947c681e6c2a72b05e6ca P a04a37fc4aa7460cbfdbeeb4e451e1f5: No bootstrap required, opened a new log
I20260925 03:03:53.494382 24662 ts_tablet_manager.cc:1397] T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c: Time spent bootstrapping tablet: real 0.128s user 0.017s sys 0.027s
I20260925 03:03:53.496796 24663 ts_tablet_manager.cc:1397] T c5aec38e66a947c681e6c2a72b05e6ca P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent bootstrapping tablet: real 0.072s user 0.019s sys 0.007s
I20260925 03:03:53.496968 24662 raft_consensus.cc:348] T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:53.497767 24662 raft_consensus.cc:374] T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:53.497658 24769 leader_election.cc:288] T 0b5df303cd6a4d59aca5c18bbbff24ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:53.497133 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b5df303cd6a4d59aca5c18bbbff24ed" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:03:53.498095 24662 raft_consensus.cc:729] T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Initialized, Role: FOLLOWER
I20260925 03:03:53.498078 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b5df303cd6a4d59aca5c18bbbff24ed" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:03:53.498756 24471 raft_consensus.cc:2454] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 0.
I20260925 03:03:53.498984 24622 raft_consensus.cc:2454] T 0b5df303cd6a4d59aca5c18bbbff24ed P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 0.
I20260925 03:03:53.499190 24662 consensus_queue.cc:260] T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:53.500320 24514 leader_election.cc:302] T 0b5df303cd6a4d59aca5c18bbbff24ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:03:53.501153 24769 raft_consensus.cc:2781] T 0b5df303cd6a4d59aca5c18bbbff24ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 03:03:53.501492 24769 raft_consensus.cc:482] T 0b5df303cd6a4d59aca5c18bbbff24ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:03:53.502009 24769 raft_consensus.cc:3037] T 0b5df303cd6a4d59aca5c18bbbff24ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:53.505468 24662 ts_tablet_manager.cc:1428] T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s
I20260925 03:03:53.506477 24662 tablet_bootstrap.cc:492] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c: Bootstrap starting.
I20260925 03:03:53.510792 24471 raft_consensus.cc:3037] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:53.512292 24719 consensus_queue.cc:1035] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260925 03:03:53.530328 24693 raft_consensus.cc:482] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:53.530829 24693 raft_consensus.cc:504] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:53.532454 24769 raft_consensus.cc:504] T 0b5df303cd6a4d59aca5c18bbbff24ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:53.532898 24693 leader_election.cc:288] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:53.534670 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5332a5d329493991dff994e70ea92c" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:03:53.543546 24769 leader_election.cc:288] T 0b5df303cd6a4d59aca5c18bbbff24ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
W20260925 03:03:53.544291 24511 leader_election.cc:341] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:53.545462 24664 tablet_bootstrap.cc:492] T 4e5332a5d329493991dff994e70ea92c P 869b0a8575b54d08ae2817fad5dd42d5: No bootstrap required, opened a new log
I20260925 03:03:53.545943 24664 ts_tablet_manager.cc:1397] T 4e5332a5d329493991dff994e70ea92c P 869b0a8575b54d08ae2817fad5dd42d5: Time spent bootstrapping tablet: real 0.076s user 0.031s sys 0.020s
I20260925 03:03:53.546013 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b5df303cd6a4d59aca5c18bbbff24ed" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:03:53.546657 24619 raft_consensus.cc:3037] T 0b5df303cd6a4d59aca5c18bbbff24ed P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:53.549050 24664 raft_consensus.cc:348] T 4e5332a5d329493991dff994e70ea92c P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:53.549787 24664 raft_consensus.cc:374] T 4e5332a5d329493991dff994e70ea92c P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:53.549453 24663 raft_consensus.cc:348] T c5aec38e66a947c681e6c2a72b05e6ca P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:53.550947 24663 raft_consensus.cc:374] T c5aec38e66a947c681e6c2a72b05e6ca P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:53.551296 24663 raft_consensus.cc:729] T c5aec38e66a947c681e6c2a72b05e6ca P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Initialized, Role: FOLLOWER
I20260925 03:03:53.549916 24717 consensus_queue.cc:1035] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260925 03:03:53.550105 24664 raft_consensus.cc:729] T 4e5332a5d329493991dff994e70ea92c P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Initialized, Role: FOLLOWER
I20260925 03:03:53.534475 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5332a5d329493991dff994e70ea92c" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:03:53.550618 24662 tablet_bootstrap.cc:654] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:53.536960 24693 raft_consensus.cc:482] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:53.552789 24664 consensus_queue.cc:260] T 4e5332a5d329493991dff994e70ea92c P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:53.552088 24663 consensus_queue.cc:260] T c5aec38e66a947c681e6c2a72b05e6ca P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:53.557061 24473 raft_consensus.cc:2454] T 4e5332a5d329493991dff994e70ea92c P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 0.
I20260925 03:03:53.553640 24693 raft_consensus.cc:504] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:53.557312 24664 ts_tablet_manager.cc:1428] T 4e5332a5d329493991dff994e70ea92c P 869b0a8575b54d08ae2817fad5dd42d5: Time spent starting tablet: real 0.011s user 0.008s sys 0.000s
I20260925 03:03:53.558418 24664 tablet_bootstrap.cc:492] T 43366a0a2f9d4344bf4e37ffd290ef8a P 869b0a8575b54d08ae2817fad5dd42d5: Bootstrap starting.
I20260925 03:03:53.559566 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b627ecb4a5384ef392aa695e06599c39" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:03:53.560266 24472 raft_consensus.cc:2454] T b627ecb4a5384ef392aa695e06599c39 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 0.
I20260925 03:03:53.561596 24514 leader_election.cc:302] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:03:53.563388 24619 raft_consensus.cc:2454] T 0b5df303cd6a4d59aca5c18bbbff24ed P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 1.
I20260925 03:03:53.563727 24769 raft_consensus.cc:2781] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 03:03:53.564118 24769 raft_consensus.cc:482] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:03:53.564452 24769 raft_consensus.cc:3037] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:53.567090 24664 tablet_bootstrap.cc:654] T 43366a0a2f9d4344bf4e37ffd290ef8a P 869b0a8575b54d08ae2817fad5dd42d5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:53.569281 24693 leader_election.cc:288] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:53.554558 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b5df303cd6a4d59aca5c18bbbff24ed" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:03:53.570403 24471 raft_consensus.cc:3037] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:53.567648 24663 ts_tablet_manager.cc:1428] T c5aec38e66a947c681e6c2a72b05e6ca P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent starting tablet: real 0.070s user 0.006s sys 0.000s
I20260925 03:03:53.573515 24663 tablet_bootstrap.cc:492] T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Bootstrap starting.
I20260925 03:03:53.577765 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b627ecb4a5384ef392aa695e06599c39" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:03:53.578506 24620 raft_consensus.cc:2454] T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 0.
I20260925 03:03:53.580550 24664 tablet_bootstrap.cc:492] T 43366a0a2f9d4344bf4e37ffd290ef8a P 869b0a8575b54d08ae2817fad5dd42d5: No bootstrap required, opened a new log
I20260925 03:03:53.572554 24769 raft_consensus.cc:504] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:53.581117 24664 ts_tablet_manager.cc:1397] T 43366a0a2f9d4344bf4e37ffd290ef8a P 869b0a8575b54d08ae2817fad5dd42d5: Time spent bootstrapping tablet: real 0.023s user 0.012s sys 0.005s
I20260925 03:03:53.583693 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b627ecb4a5384ef392aa695e06599c39" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:03:53.584267 24620 raft_consensus.cc:3037] T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:53.584627 24664 raft_consensus.cc:348] T 43366a0a2f9d4344bf4e37ffd290ef8a P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:53.585505 24664 raft_consensus.cc:374] T 43366a0a2f9d4344bf4e37ffd290ef8a P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:53.585812 24664 raft_consensus.cc:729] T 43366a0a2f9d4344bf4e37ffd290ef8a P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Initialized, Role: FOLLOWER
I20260925 03:03:53.586611 24664 consensus_queue.cc:260] T 43366a0a2f9d4344bf4e37ffd290ef8a P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:53.588485 24711 consensus_queue.cc:1035] T c85f5d1d3f004cf5817bb594fd8e6469 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 03:03:53.589150 24664 ts_tablet_manager.cc:1428] T 43366a0a2f9d4344bf4e37ffd290ef8a P 869b0a8575b54d08ae2817fad5dd42d5: Time spent starting tablet: real 0.008s user 0.005s sys 0.001s
I20260925 03:03:53.590174 24664 tablet_bootstrap.cc:492] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P 869b0a8575b54d08ae2817fad5dd42d5: Bootstrap starting.
I20260925 03:03:53.596191 24664 tablet_bootstrap.cc:654] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P 869b0a8575b54d08ae2817fad5dd42d5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:53.595836 24763 consensus_queue.cc:1035] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 03:03:53.596033 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b627ecb4a5384ef392aa695e06599c39" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:03:53.596851 24470 raft_consensus.cc:3037] T b627ecb4a5384ef392aa695e06599c39 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:53.601259 24620 raft_consensus.cc:2454] T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 1.
I20260925 03:03:53.602342 24511 leader_election.cc:302] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:03:53.602775 24769 leader_election.cc:288] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:53.606524 24693 raft_consensus.cc:2781] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:03:53.607220 24693 raft_consensus.cc:686] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 LEADER]: Becoming Leader. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:03:53.611279 24663 tablet_bootstrap.cc:654] T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:53.615577 24514 leader_election.cc:302] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:03:53.618423 24717 raft_consensus.cc:482] T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:53.618719 24511 leader_election.cc:302] T 0b5df303cd6a4d59aca5c18bbbff24ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:03:53.619550 24763 raft_consensus.cc:482] T c5aec38e66a947c681e6c2a72b05e6ca P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:53.620234 24763 raft_consensus.cc:504] T c5aec38e66a947c681e6c2a72b05e6ca P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:53.621713 24770 raft_consensus.cc:2781] T 0b5df303cd6a4d59aca5c18bbbff24ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:03:53.622162 24770 raft_consensus.cc:686] T 0b5df303cd6a4d59aca5c18bbbff24ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 LEADER]: Becoming Leader. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:03:53.632458 24770 consensus_queue.cc:237] T 0b5df303cd6a4d59aca5c18bbbff24ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:53.635993 24813 raft_consensus.cc:2781] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 03:03:53.618927 24717 raft_consensus.cc:504] T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:53.636039 24693 consensus_queue.cc:237] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:03:53.643692 24436 consensus_peers.cc:487] T c85f5d1d3f004cf5817bb594fd8e6469 P 869b0a8575b54d08ae2817fad5dd42d5 -> Peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105): Couldn't send request to peer eeab93abfdb141a895002ec9f312296c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260925 03:03:53.647857 24470 raft_consensus.cc:2454] T b627ecb4a5384ef392aa695e06599c39 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 1.
I20260925 03:03:53.649324 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5aec38e66a947c681e6c2a72b05e6ca" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:03:53.637523 24473 raft_consensus.cc:3037] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:53.637482 24813 raft_consensus.cc:482] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:03:53.641644 24763 leader_election.cc:288] T c5aec38e66a947c681e6c2a72b05e6ca P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:53.656042 24471 raft_consensus.cc:2454] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 1.
I20260925 03:03:53.654868 24717 leader_election.cc:288] T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:03:53.656978 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06f4a7eaf6534e4986c9fd35911038e4" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:03:53.654561 24813 raft_consensus.cc:3037] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:53.662590 24769 consensus_queue.cc:1035] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 03:03:53.663285 24622 raft_consensus.cc:3037] T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:53.668666 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5aec38e66a947c681e6c2a72b05e6ca" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:03:53.669644 24619 raft_consensus.cc:2379] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 1: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:03:53.670674 24511 leader_election.cc:302] T c5aec38e66a947c681e6c2a72b05e6ca P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:03:53.678117 24693 raft_consensus.cc:3037] T c5aec38e66a947c681e6c2a72b05e6ca P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:53.681618 24663 tablet_bootstrap.cc:492] T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5: No bootstrap required, opened a new log
I20260925 03:03:53.682143 24663 ts_tablet_manager.cc:1397] T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent bootstrapping tablet: real 0.109s user 0.007s sys 0.024s
I20260925 03:03:53.682478 24813 raft_consensus.cc:504] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:53.695151 24813 leader_election.cc:288] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:53.701174 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5332a5d329493991dff994e70ea92c" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:03:53.701803 24664 tablet_bootstrap.cc:492] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P 869b0a8575b54d08ae2817fad5dd42d5: No bootstrap required, opened a new log
I20260925 03:03:53.702316 24664 ts_tablet_manager.cc:1397] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P 869b0a8575b54d08ae2817fad5dd42d5: Time spent bootstrapping tablet: real 0.112s user 0.015s sys 0.035s
I20260925 03:03:53.692462 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06f4a7eaf6534e4986c9fd35911038e4" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
W20260925 03:03:53.710867 24511 leader_election.cc:341] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:53.712179 24588 leader_election.cc:302] T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:03:53.715605 24693 raft_consensus.cc:2726] T c5aec38e66a947c681e6c2a72b05e6ca P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 03:03:53.749322 24663 raft_consensus.cc:348] T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:53.751201 24663 raft_consensus.cc:374] T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:53.745669 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5332a5d329493991dff994e70ea92c" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:03:53.753794 24470 raft_consensus.cc:3037] T 4e5332a5d329493991dff994e70ea92c P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:53.755580 24370 catalog_manager.cc:5495] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2). New cstate: current_term: 1 leader_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: UNKNOWN } } }
I20260925 03:03:53.757679 24370 catalog_manager.cc:5495] T 0b5df303cd6a4d59aca5c18bbbff24ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2). New cstate: current_term: 1 leader_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: UNKNOWN } } }
I20260925 03:03:53.751457 24663 raft_consensus.cc:729] T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Initialized, Role: FOLLOWER
I20260925 03:03:53.760663 24664 raft_consensus.cc:348] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:53.761579 24664 raft_consensus.cc:374] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:53.761850 24664 raft_consensus.cc:729] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Initialized, Role: FOLLOWER
I20260925 03:03:53.762109 24663 consensus_queue.cc:260] T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:53.762362 24718 consensus_queue.cc:1035] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260925 03:03:53.763062 24664 consensus_queue.cc:260] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:53.766670 24663 ts_tablet_manager.cc:1428] T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent starting tablet: real 0.084s user 0.007s sys 0.000s
I20260925 03:03:53.767441 24662 tablet_bootstrap.cc:492] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c: No bootstrap required, opened a new log
I20260925 03:03:53.767887 24662 ts_tablet_manager.cc:1397] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c: Time spent bootstrapping tablet: real 0.262s user 0.020s sys 0.032s
I20260925 03:03:53.767611 24663 tablet_bootstrap.cc:492] T 43366a0a2f9d4344bf4e37ffd290ef8a P a04a37fc4aa7460cbfdbeeb4e451e1f5: Bootstrap starting.
I20260925 03:03:53.775205 24664 ts_tablet_manager.cc:1428] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P 869b0a8575b54d08ae2817fad5dd42d5: Time spent starting tablet: real 0.073s user 0.005s sys 0.004s
I20260925 03:03:53.776276 24664 tablet_bootstrap.cc:492] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5: Bootstrap starting.
I20260925 03:03:53.781453 24820 raft_consensus.cc:482] T 8ac466793d094325ace224708b2158ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:53.782354 24820 raft_consensus.cc:504] T 8ac466793d094325ace224708b2158ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:53.787935 24718 raft_consensus.cc:482] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:53.789345 24662 raft_consensus.cc:348] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:53.790222 24662 raft_consensus.cc:374] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:53.788730 24718 raft_consensus.cc:504] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:53.791501 24470 raft_consensus.cc:2454] T 4e5332a5d329493991dff994e70ea92c P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 1.
I20260925 03:03:53.793648 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ac466793d094325ace224708b2158ed" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:03:53.793815 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ac466793d094325ace224708b2158ed" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:03:53.794859 24547 raft_consensus.cc:2454] T 8ac466793d094325ace224708b2158ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 0.
I20260925 03:03:53.795912 24586 leader_election.cc:302] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:03:53.796767 24670 raft_consensus.cc:2781] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 03:03:53.797096 24670 raft_consensus.cc:482] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:03:53.797394 24670 raft_consensus.cc:3037] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:53.800218 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ac466793d094325ace224708b2158ed" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:03:53.800419 24718 leader_election.cc:288] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
W20260925 03:03:53.801044 24588 leader_election.cc:341] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:53.801530 24820 leader_election.cc:288] T 8ac466793d094325ace224708b2158ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:53.790486 24662 raft_consensus.cc:729] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Initialized, Role: FOLLOWER
I20260925 03:03:53.803537 24662 consensus_queue.cc:260] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:53.802584 24514 leader_election.cc:302] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:03:53.803375 24711 raft_consensus.cc:482] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:53.806583 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ac466793d094325ace224708b2158ed" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:03:53.807077 24711 raft_consensus.cc:504] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:53.808450 24820 raft_consensus.cc:2781] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader election won for term 1
W20260925 03:03:53.808904 24514 leader_election.cc:341] T 8ac466793d094325ace224708b2158ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:53.809029 24820 raft_consensus.cc:686] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 LEADER]: Becoming Leader. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:03:53.810142 24820 consensus_queue.cc:237] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:53.805608 24718 consensus_queue.cc:1035] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.010s
I20260925 03:03:53.818539 24670 raft_consensus.cc:504] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:53.819829 24620 raft_consensus.cc:2379] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 1: Already voted for candidate eeab93abfdb141a895002ec9f312296c in this term.
I20260925 03:03:53.856925 24662 ts_tablet_manager.cc:1428] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c: Time spent starting tablet: real 0.089s user 0.004s sys 0.005s
I20260925 03:03:53.858124 24662 tablet_bootstrap.cc:492] T c85f5d1d3f004cf5817bb594fd8e6469 P eeab93abfdb141a895002ec9f312296c: Bootstrap starting.
I20260925 03:03:53.882164 24693 raft_consensus.cc:482] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:53.882719 24693 raft_consensus.cc:504] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:53.889987 24664 tablet_bootstrap.cc:654] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:53.894495 24670 leader_election.cc:288] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 1 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:03:53.896250 24717 raft_consensus.cc:2726] T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 03:03:53.897542 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ac466793d094325ace224708b2158ed" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:03:53.898290 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ac466793d094325ace224708b2158ed" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
W20260925 03:03:53.898975 24588 leader_election.cc:341] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:53.898185 24693 leader_election.cc:288] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:53.892076 24718 raft_consensus.cc:482] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:53.900852 24547 raft_consensus.cc:3037] T 8ac466793d094325ace224708b2158ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:53.899562 24771 consensus_queue.cc:1035] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 03:03:53.904433 24718 raft_consensus.cc:504] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:53.917155 24619 raft_consensus.cc:3037] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:53.918154 24663 tablet_bootstrap.cc:654] T 43366a0a2f9d4344bf4e37ffd290ef8a P a04a37fc4aa7460cbfdbeeb4e451e1f5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:53.923807 24718 leader_election.cc:288] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:03:53.924579 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ac285175464fbda67211d9574db415" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:03:53.925545 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ac285175464fbda67211d9574db415" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:03:53.926331 24546 raft_consensus.cc:2379] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c in current term 1: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:03:53.914376 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60d45b7d467d4e1ab92367f6d7d52b15" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:03:53.928015 24470 raft_consensus.cc:2454] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 0.
I20260925 03:03:53.929270 24514 leader_election.cc:302] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:03:53.930316 24814 raft_consensus.cc:2781] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 03:03:53.930675 24814 raft_consensus.cc:482] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:03:53.930971 24814 raft_consensus.cc:3037] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:53.935356 24586 leader_election.cc:302] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:03:53.936386 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:03:53.936533 24718 raft_consensus.cc:2726] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 03:03:53.937949 24711 leader_election.cc:288] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:53.949337 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60d45b7d467d4e1ab92367f6d7d52b15" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
W20260925 03:03:53.953794 24511 leader_election.cc:341] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:53.954682 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:03:53.955940 24662 tablet_bootstrap.cc:654] T c85f5d1d3f004cf5817bb594fd8e6469 P eeab93abfdb141a895002ec9f312296c: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:53.947968 24671 consensus_queue.cc:1035] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.010s
I20260925 03:03:53.946182 24711 consensus_queue.cc:1035] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.012s
I20260925 03:03:53.970671 24511 leader_election.cc:302] T 8ac466793d094325ace224708b2158ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:03:53.954741 24814 raft_consensus.cc:504] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:53.975464 24708 consensus_queue.cc:1035] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 03:03:53.988703 24547 raft_consensus.cc:2454] T 8ac466793d094325ace224708b2158ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 1.
I20260925 03:03:53.989306 24693 raft_consensus.cc:2726] T 8ac466793d094325ace224708b2158ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 03:03:53.991266 24370 catalog_manager.cc:5495] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2). New cstate: current_term: 1 leader_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: UNKNOWN } } }
I20260925 03:03:53.992623 24586 leader_election.cc:302] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:03:53.993727 24718 raft_consensus.cc:2781] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:03:53.994191 24718 raft_consensus.cc:686] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [term 1 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
W20260925 03:03:53.999886 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 49710.3us
I20260925 03:03:54.000816 24437 leader_election.cc:302] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:03:54.007426 24664 tablet_bootstrap.cc:492] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5: No bootstrap required, opened a new log
I20260925 03:03:54.008054 24664 ts_tablet_manager.cc:1397] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5: Time spent bootstrapping tablet: real 0.232s user 0.014s sys 0.021s
I20260925 03:03:54.013264 24718 consensus_queue.cc:237] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:54.015517 24663 tablet_bootstrap.cc:492] T 43366a0a2f9d4344bf4e37ffd290ef8a P a04a37fc4aa7460cbfdbeeb4e451e1f5: No bootstrap required, opened a new log
I20260925 03:03:54.016587 24663 ts_tablet_manager.cc:1397] T 43366a0a2f9d4344bf4e37ffd290ef8a P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent bootstrapping tablet: real 0.249s user 0.013s sys 0.014s
I20260925 03:03:54.015290 24708 raft_consensus.cc:2726] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 03:03:54.026646 24664 raft_consensus.cc:348] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:54.031245 24771 raft_consensus.cc:482] T 43366a0a2f9d4344bf4e37ffd290ef8a P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:54.033113 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60d45b7d467d4e1ab92367f6d7d52b15" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:03:54.033922 24472 raft_consensus.cc:3037] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:54.034268 24771 raft_consensus.cc:504] T 43366a0a2f9d4344bf4e37ffd290ef8a P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:54.032945 24664 raft_consensus.cc:374] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:54.037518 24664 raft_consensus.cc:729] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Initialized, Role: FOLLOWER
I20260925 03:03:54.038141 24664 consensus_queue.cc:260] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:54.038082 24814 leader_election.cc:288] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:54.042307 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60d45b7d467d4e1ab92367f6d7d52b15" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:03:54.049431 24664 ts_tablet_manager.cc:1428] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5: Time spent starting tablet: real 0.041s user 0.006s sys 0.002s
I20260925 03:03:54.050099 24771 leader_election.cc:288] T 43366a0a2f9d4344bf4e37ffd290ef8a P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
W20260925 03:03:54.050638 24511 leader_election.cc:341] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:54.051415 24820 raft_consensus.cc:482] T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:54.057265 24820 raft_consensus.cc:504] T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:54.059737 24820 leader_election.cc:288] T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:54.062217 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43366a0a2f9d4344bf4e37ffd290ef8a" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:03:54.063982 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e8af2641e124f9589c5ccfdac785f63" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:03:54.064250 24472 raft_consensus.cc:2454] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 1.
W20260925 03:03:54.065245 24511 leader_election.cc:341] T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:54.051604 24664 tablet_bootstrap.cc:492] T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5: Bootstrap starting.
I20260925 03:03:54.050882 24663 raft_consensus.cc:348] T 43366a0a2f9d4344bf4e37ffd290ef8a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:54.052352 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43366a0a2f9d4344bf4e37ffd290ef8a" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:03:54.067071 24663 raft_consensus.cc:374] T 43366a0a2f9d4344bf4e37ffd290ef8a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:54.067395 24663 raft_consensus.cc:729] T 43366a0a2f9d4344bf4e37ffd290ef8a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Initialized, Role: FOLLOWER
I20260925 03:03:54.068606 24663 consensus_queue.cc:260] T 43366a0a2f9d4344bf4e37ffd290ef8a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:54.071592 24546 raft_consensus.cc:2454] T 43366a0a2f9d4344bf4e37ffd290ef8a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 0.
I20260925 03:03:54.069101 24514 leader_election.cc:302] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:03:54.075251 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e8af2641e124f9589c5ccfdac785f63" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:03:54.076325 24662 tablet_bootstrap.cc:492] T c85f5d1d3f004cf5817bb594fd8e6469 P eeab93abfdb141a895002ec9f312296c: No bootstrap required, opened a new log
I20260925 03:03:54.069522 24820 consensus_queue.cc:1035] T 0b5df303cd6a4d59aca5c18bbbff24ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20260925 03:03:54.076798 24662 ts_tablet_manager.cc:1397] T c85f5d1d3f004cf5817bb594fd8e6469 P eeab93abfdb141a895002ec9f312296c: Time spent bootstrapping tablet: real 0.219s user 0.022s sys 0.011s
I20260925 03:03:54.076910 24814 raft_consensus.cc:2781] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader election won for term 1
W20260925 03:03:54.071640 24436 leader_election.cc:341] T 43366a0a2f9d4344bf4e37ffd290ef8a P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105): Illegal state: must be running to vote when last-logged opid is not known
W20260925 03:03:54.081205 24514 leader_election.cc:341] T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:54.081691 24514 leader_election.cc:302] T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:03:54.081898 24693 consensus_queue.cc:1035] T 0b5df303cd6a4d59aca5c18bbbff24ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 03:03:54.082461 24664 tablet_bootstrap.cc:654] T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:54.082444 24820 raft_consensus.cc:2726] T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 03:03:54.077333 24814 raft_consensus.cc:686] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 LEADER]: Becoming Leader. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:03:54.085109 24437 leader_election.cc:302] T 43366a0a2f9d4344bf4e37ffd290ef8a P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:03:54.085186 24814 consensus_queue.cc:237] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:54.086017 24771 raft_consensus.cc:2781] T 43366a0a2f9d4344bf4e37ffd290ef8a P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 03:03:54.089686 24771 raft_consensus.cc:482] T 43366a0a2f9d4344bf4e37ffd290ef8a P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:03:54.090374 24771 raft_consensus.cc:3037] T 43366a0a2f9d4344bf4e37ffd290ef8a P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:54.107846 24369 catalog_manager.cc:5495] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c reported cstate change: term changed from 0 to 1, leader changed from <none> to eeab93abfdb141a895002ec9f312296c (127.22.193.3). New cstate: current_term: 1 leader_uuid: "eeab93abfdb141a895002ec9f312296c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: HEALTHY } } }
I20260925 03:03:54.108932 24663 ts_tablet_manager.cc:1428] T 43366a0a2f9d4344bf4e37ffd290ef8a P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent starting tablet: real 0.092s user 0.010s sys 0.000s
I20260925 03:03:54.109860 24663 tablet_bootstrap.cc:492] T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Bootstrap starting.
I20260925 03:03:54.110015 24662 raft_consensus.cc:348] T c85f5d1d3f004cf5817bb594fd8e6469 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:54.110595 24662 raft_consensus.cc:374] T c85f5d1d3f004cf5817bb594fd8e6469 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:54.110800 24662 raft_consensus.cc:729] T c85f5d1d3f004cf5817bb594fd8e6469 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Initialized, Role: FOLLOWER
I20260925 03:03:54.111378 24662 consensus_queue.cc:260] T c85f5d1d3f004cf5817bb594fd8e6469 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:54.115470 24708 raft_consensus.cc:482] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:54.115953 24708 raft_consensus.cc:504] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:54.118968 24708 leader_election.cc:288] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:54.120585 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5278a60f97da4e10aecd50a8d9a9a43c" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:03:54.127796 24662 ts_tablet_manager.cc:1428] T c85f5d1d3f004cf5817bb594fd8e6469 P eeab93abfdb141a895002ec9f312296c: Time spent starting tablet: real 0.050s user 0.006s sys 0.000s
I20260925 03:03:54.128944 24662 tablet_bootstrap.cc:492] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c: Bootstrap starting.
W20260925 03:03:54.135154 24436 leader_election.cc:341] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:54.138787 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5278a60f97da4e10aecd50a8d9a9a43c" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:03:54.125664 24771 raft_consensus.cc:504] T 43366a0a2f9d4344bf4e37ffd290ef8a P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:03:54.143713 24437 leader_election.cc:341] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:54.147957 24771 leader_election.cc:288] T 43366a0a2f9d4344bf4e37ffd290ef8a P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:54.147660 24662 tablet_bootstrap.cc:654] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:54.149379 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43366a0a2f9d4344bf4e37ffd290ef8a" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:03:54.147073 24437 leader_election.cc:302] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:03:54.154350 24664 tablet_bootstrap.cc:492] T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5: No bootstrap required, opened a new log
I20260925 03:03:54.155272 24664 ts_tablet_manager.cc:1397] T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5: Time spent bootstrapping tablet: real 0.104s user 0.010s sys 0.004s
W20260925 03:03:54.153787 24436 leader_election.cc:341] T 43366a0a2f9d4344bf4e37ffd290ef8a P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:54.157902 24771 raft_consensus.cc:2726] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 03:03:54.160778 24369 catalog_manager.cc:5495] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2). New cstate: current_term: 1 leader_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: UNKNOWN } } }
I20260925 03:03:54.167483 24771 raft_consensus.cc:482] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:54.168090 24771 raft_consensus.cc:504] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:54.171902 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abd5b8dc1b7e49e3aa9af900d73ff0e7" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
W20260925 03:03:54.173550 24436 leader_election.cc:341] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:54.174351 24663 tablet_bootstrap.cc:654] T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:54.174443 24771 leader_election.cc:288] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:54.176445 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abd5b8dc1b7e49e3aa9af900d73ff0e7" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:03:54.179325 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43366a0a2f9d4344bf4e37ffd290ef8a" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:03:54.180239 24547 raft_consensus.cc:3037] T 43366a0a2f9d4344bf4e37ffd290ef8a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Advancing to term 1
W20260925 03:03:54.182785 24437 leader_election.cc:341] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:54.183358 24437 leader_election.cc:302] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:03:54.184823 24771 raft_consensus.cc:2726] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 03:03:54.188066 24770 consensus_queue.cc:1035] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260925 03:03:54.188750 24662 tablet_bootstrap.cc:492] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c: No bootstrap required, opened a new log
I20260925 03:03:54.189347 24662 ts_tablet_manager.cc:1397] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c: Time spent bootstrapping tablet: real 0.061s user 0.009s sys 0.005s
I20260925 03:03:54.191967 24662 raft_consensus.cc:348] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:54.192657 24662 raft_consensus.cc:374] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:54.192950 24662 raft_consensus.cc:729] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Initialized, Role: FOLLOWER
I20260925 03:03:54.193580 24662 consensus_queue.cc:260] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:54.194201 24763 consensus_queue.cc:1035] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 03:03:54.195770 24662 ts_tablet_manager.cc:1428] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c: Time spent starting tablet: real 0.006s user 0.001s sys 0.003s
I20260925 03:03:54.197175 24662 tablet_bootstrap.cc:492] T 43366a0a2f9d4344bf4e37ffd290ef8a P eeab93abfdb141a895002ec9f312296c: Bootstrap starting.
I20260925 03:03:54.201946 24547 raft_consensus.cc:2454] T 43366a0a2f9d4344bf4e37ffd290ef8a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 1.
I20260925 03:03:54.203133 24437 leader_election.cc:302] T 43366a0a2f9d4344bf4e37ffd290ef8a P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:03:54.204573 24708 raft_consensus.cc:2781] T 43366a0a2f9d4344bf4e37ffd290ef8a P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:03:54.205091 24708 raft_consensus.cc:686] T 43366a0a2f9d4344bf4e37ffd290ef8a P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 LEADER]: Becoming Leader. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:03:54.205943 24708 consensus_queue.cc:237] T 43366a0a2f9d4344bf4e37ffd290ef8a P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:54.222997 24662 tablet_bootstrap.cc:654] T 43366a0a2f9d4344bf4e37ffd290ef8a P eeab93abfdb141a895002ec9f312296c: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:54.261734 24371 catalog_manager.cc:5495] T 43366a0a2f9d4344bf4e37ffd290ef8a P 869b0a8575b54d08ae2817fad5dd42d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1). New cstate: current_term: 1 leader_uuid: "869b0a8575b54d08ae2817fad5dd42d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: UNKNOWN } } }
I20260925 03:03:54.264822 24664 raft_consensus.cc:348] T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:54.265704 24664 raft_consensus.cc:374] T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:54.265985 24664 raft_consensus.cc:729] T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Initialized, Role: FOLLOWER
I20260925 03:03:54.271196 24664 consensus_queue.cc:260] T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:54.278771 24664 ts_tablet_manager.cc:1428] T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5: Time spent starting tablet: real 0.123s user 0.007s sys 0.001s
I20260925 03:03:54.279886 24664 tablet_bootstrap.cc:492] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5: Bootstrap starting.
I20260925 03:03:54.283658 24708 raft_consensus.cc:482] T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:54.284190 24708 raft_consensus.cc:504] T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:54.287770 24664 tablet_bootstrap.cc:654] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:54.290060 24708 leader_election.cc:288] T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:54.292085 24662 tablet_bootstrap.cc:492] T 43366a0a2f9d4344bf4e37ffd290ef8a P eeab93abfdb141a895002ec9f312296c: No bootstrap required, opened a new log
I20260925 03:03:54.293713 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74443a0a1dc34d88a2f42e3b722db405" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:03:54.301752 24662 ts_tablet_manager.cc:1397] T 43366a0a2f9d4344bf4e37ffd290ef8a P eeab93abfdb141a895002ec9f312296c: Time spent bootstrapping tablet: real 0.105s user 0.017s sys 0.030s
I20260925 03:03:54.294189 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74443a0a1dc34d88a2f42e3b722db405" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:03:54.294741 24820 consensus_queue.cc:1035] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 03:03:54.303720 24436 leader_election.cc:341] T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105): Illegal state: must be running to vote when last-logged opid is not known
W20260925 03:03:54.308125 24437 leader_election.cc:341] T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:54.308005 24662 raft_consensus.cc:348] T 43366a0a2f9d4344bf4e37ffd290ef8a P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:54.308965 24662 raft_consensus.cc:374] T 43366a0a2f9d4344bf4e37ffd290ef8a P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:54.308830 24437 leader_election.cc:302] T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:03:54.309280 24662 raft_consensus.cc:729] T 43366a0a2f9d4344bf4e37ffd290ef8a P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Initialized, Role: FOLLOWER
I20260925 03:03:54.309947 24771 raft_consensus.cc:2726] T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 03:03:54.310057 24662 consensus_queue.cc:260] T 43366a0a2f9d4344bf4e37ffd290ef8a P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:54.311602 24663 tablet_bootstrap.cc:492] T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5: No bootstrap required, opened a new log
I20260925 03:03:54.312069 24663 ts_tablet_manager.cc:1397] T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent bootstrapping tablet: real 0.202s user 0.056s sys 0.073s
I20260925 03:03:54.312573 24662 ts_tablet_manager.cc:1428] T 43366a0a2f9d4344bf4e37ffd290ef8a P eeab93abfdb141a895002ec9f312296c: Time spent starting tablet: real 0.010s user 0.005s sys 0.001s
I20260925 03:03:54.313050 24718 raft_consensus.cc:482] T c85f5d1d3f004cf5817bb594fd8e6469 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:54.313483 24718 raft_consensus.cc:504] T c85f5d1d3f004cf5817bb594fd8e6469 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:54.314468 24663 raft_consensus.cc:348] T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:54.315142 24663 raft_consensus.cc:374] T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:54.315390 24663 raft_consensus.cc:729] T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Initialized, Role: FOLLOWER
I20260925 03:03:54.316023 24663 consensus_queue.cc:260] T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:54.316718 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c85f5d1d3f004cf5817bb594fd8e6469" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:03:54.318221 24663 ts_tablet_manager.cc:1428] T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 03:03:54.319154 24663 tablet_bootstrap.cc:492] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Bootstrap starting.
I20260925 03:03:54.320708 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c85f5d1d3f004cf5817bb594fd8e6469" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:03:54.323144 24588 leader_election.cc:302] T c85f5d1d3f004cf5817bb594fd8e6469 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:03:54.324016 24719 raft_consensus.cc:2726] T c85f5d1d3f004cf5817bb594fd8e6469 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 03:03:54.325343 24663 tablet_bootstrap.cc:654] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:54.315399 24718 leader_election.cc:288] T c85f5d1d3f004cf5817bb594fd8e6469 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:03:54.327176 24664 tablet_bootstrap.cc:492] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5: No bootstrap required, opened a new log
I20260925 03:03:54.329092 24662 tablet_bootstrap.cc:492] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c: Bootstrap starting.
I20260925 03:03:54.327617 24664 ts_tablet_manager.cc:1397] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5: Time spent bootstrapping tablet: real 0.048s user 0.010s sys 0.001s
I20260925 03:03:54.339044 24662 tablet_bootstrap.cc:654] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:54.344470 24664 raft_consensus.cc:348] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:54.345321 24664 raft_consensus.cc:374] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:54.345597 24664 raft_consensus.cc:729] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Initialized, Role: FOLLOWER
I20260925 03:03:54.346267 24664 consensus_queue.cc:260] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:03:54.347609 24511 consensus_peers.cc:487] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5 -> Peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105): Couldn't send request to peer eeab93abfdb141a895002ec9f312296c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260925 03:03:54.355206 24662 tablet_bootstrap.cc:492] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c: No bootstrap required, opened a new log
I20260925 03:03:54.356433 24662 ts_tablet_manager.cc:1397] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c: Time spent bootstrapping tablet: real 0.028s user 0.013s sys 0.005s
I20260925 03:03:54.358637 24718 raft_consensus.cc:482] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:54.359226 24718 raft_consensus.cc:504] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:54.359889 24664 ts_tablet_manager.cc:1428] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5: Time spent starting tablet: real 0.029s user 0.003s sys 0.002s
I20260925 03:03:54.359900 24662 raft_consensus.cc:348] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:54.360792 24662 raft_consensus.cc:374] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:54.360908 24664 tablet_bootstrap.cc:492] T b4ebd2b043024a548005c8b6b00b0449 P 869b0a8575b54d08ae2817fad5dd42d5: Bootstrap starting.
I20260925 03:03:54.361102 24662 raft_consensus.cc:729] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Initialized, Role: FOLLOWER
I20260925 03:03:54.361387 24718 leader_election.cc:288] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:03:54.361792 24662 consensus_queue.cc:260] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:54.364030 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60d45b7d467d4e1ab92367f6d7d52b15" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:03:54.364406 24662 ts_tablet_manager.cc:1428] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c: Time spent starting tablet: real 0.008s user 0.003s sys 0.003s
I20260925 03:03:54.364993 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60d45b7d467d4e1ab92367f6d7d52b15" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:03:54.365412 24662 tablet_bootstrap.cc:492] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c: Bootstrap starting.
I20260925 03:03:54.365880 24472 raft_consensus.cc:2379] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c in current term 1: Already voted for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in this term.
I20260925 03:03:54.367941 24664 tablet_bootstrap.cc:654] T b4ebd2b043024a548005c8b6b00b0449 P 869b0a8575b54d08ae2817fad5dd42d5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:54.371353 24662 tablet_bootstrap.cc:654] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:54.372454 24586 leader_election.cc:302] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:03:54.373844 24718 raft_consensus.cc:3037] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:54.381348 24663 tablet_bootstrap.cc:492] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P a04a37fc4aa7460cbfdbeeb4e451e1f5: No bootstrap required, opened a new log
I20260925 03:03:54.381764 24718 raft_consensus.cc:2726] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 03:03:54.381917 24663 ts_tablet_manager.cc:1397] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent bootstrapping tablet: real 0.063s user 0.031s sys 0.024s
I20260925 03:03:54.385169 24663 raft_consensus.cc:348] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:54.386085 24663 raft_consensus.cc:374] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:54.387471 24663 raft_consensus.cc:729] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Initialized, Role: FOLLOWER
I20260925 03:03:54.388182 24663 consensus_queue.cc:260] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:54.390381 24663 ts_tablet_manager.cc:1428] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent starting tablet: real 0.008s user 0.002s sys 0.004s
I20260925 03:03:54.391618 24663 tablet_bootstrap.cc:492] T a7aa47c48404465b94a36fd819fe3c7a P a04a37fc4aa7460cbfdbeeb4e451e1f5: Bootstrap starting.
I20260925 03:03:54.397145 24663 tablet_bootstrap.cc:654] T a7aa47c48404465b94a36fd819fe3c7a P a04a37fc4aa7460cbfdbeeb4e451e1f5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:54.400867 24664 tablet_bootstrap.cc:492] T b4ebd2b043024a548005c8b6b00b0449 P 869b0a8575b54d08ae2817fad5dd42d5: No bootstrap required, opened a new log
I20260925 03:03:54.401340 24664 ts_tablet_manager.cc:1397] T b4ebd2b043024a548005c8b6b00b0449 P 869b0a8575b54d08ae2817fad5dd42d5: Time spent bootstrapping tablet: real 0.041s user 0.008s sys 0.007s
I20260925 03:03:54.402069 24662 tablet_bootstrap.cc:492] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c: No bootstrap required, opened a new log
I20260925 03:03:54.402532 24662 ts_tablet_manager.cc:1397] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c: Time spent bootstrapping tablet: real 0.037s user 0.010s sys 0.016s
I20260925 03:03:54.403690 24664 raft_consensus.cc:348] T b4ebd2b043024a548005c8b6b00b0449 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:54.404412 24664 raft_consensus.cc:374] T b4ebd2b043024a548005c8b6b00b0449 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:54.404675 24664 raft_consensus.cc:729] T b4ebd2b043024a548005c8b6b00b0449 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Initialized, Role: FOLLOWER
I20260925 03:03:54.404845 24662 raft_consensus.cc:348] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:54.405535 24662 raft_consensus.cc:374] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:54.405884 24662 raft_consensus.cc:729] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Initialized, Role: FOLLOWER
I20260925 03:03:54.405486 24664 consensus_queue.cc:260] T b4ebd2b043024a548005c8b6b00b0449 P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:54.406577 24662 consensus_queue.cc:260] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:54.411589 24664 ts_tablet_manager.cc:1428] T b4ebd2b043024a548005c8b6b00b0449 P 869b0a8575b54d08ae2817fad5dd42d5: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20260925 03:03:54.412262 24663 tablet_bootstrap.cc:492] T a7aa47c48404465b94a36fd819fe3c7a P a04a37fc4aa7460cbfdbeeb4e451e1f5: No bootstrap required, opened a new log
I20260925 03:03:54.412802 24663 ts_tablet_manager.cc:1397] T a7aa47c48404465b94a36fd819fe3c7a P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent bootstrapping tablet: real 0.021s user 0.009s sys 0.011s
I20260925 03:03:54.414146 24662 ts_tablet_manager.cc:1428] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c: Time spent starting tablet: real 0.011s user 0.006s sys 0.001s
I20260925 03:03:54.415050 24662 tablet_bootstrap.cc:492] T 5278a60f97da4e10aecd50a8d9a9a43c P eeab93abfdb141a895002ec9f312296c: Bootstrap starting.
I20260925 03:03:54.415412 24663 raft_consensus.cc:348] T a7aa47c48404465b94a36fd819fe3c7a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:54.416170 24663 raft_consensus.cc:374] T a7aa47c48404465b94a36fd819fe3c7a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:54.416481 24663 raft_consensus.cc:729] T a7aa47c48404465b94a36fd819fe3c7a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Initialized, Role: FOLLOWER
I20260925 03:03:54.417248 24663 consensus_queue.cc:260] T a7aa47c48404465b94a36fd819fe3c7a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:54.421597 24662 tablet_bootstrap.cc:654] T 5278a60f97da4e10aecd50a8d9a9a43c P eeab93abfdb141a895002ec9f312296c: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:54.423491 24663 ts_tablet_manager.cc:1428] T a7aa47c48404465b94a36fd819fe3c7a P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20260925 03:03:54.412539 24664 tablet_bootstrap.cc:492] T a7aa47c48404465b94a36fd819fe3c7a P 869b0a8575b54d08ae2817fad5dd42d5: Bootstrap starting.
I20260925 03:03:54.424515 24663 tablet_bootstrap.cc:492] T a4abe0ea757e4886be4945c5001e529a P a04a37fc4aa7460cbfdbeeb4e451e1f5: Bootstrap starting.
I20260925 03:03:54.431700 24664 tablet_bootstrap.cc:654] T a7aa47c48404465b94a36fd819fe3c7a P 869b0a8575b54d08ae2817fad5dd42d5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:54.431874 24663 tablet_bootstrap.cc:654] T a4abe0ea757e4886be4945c5001e529a P a04a37fc4aa7460cbfdbeeb4e451e1f5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:54.434619 24718 consensus_queue.cc:1035] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260925 03:03:54.441747 24662 tablet_bootstrap.cc:492] T 5278a60f97da4e10aecd50a8d9a9a43c P eeab93abfdb141a895002ec9f312296c: No bootstrap required, opened a new log
I20260925 03:03:54.442165 24664 tablet_bootstrap.cc:492] T a7aa47c48404465b94a36fd819fe3c7a P 869b0a8575b54d08ae2817fad5dd42d5: No bootstrap required, opened a new log
I20260925 03:03:54.442246 24662 ts_tablet_manager.cc:1397] T 5278a60f97da4e10aecd50a8d9a9a43c P eeab93abfdb141a895002ec9f312296c: Time spent bootstrapping tablet: real 0.027s user 0.014s sys 0.001s
I20260925 03:03:54.442655 24664 ts_tablet_manager.cc:1397] T a7aa47c48404465b94a36fd819fe3c7a P 869b0a8575b54d08ae2817fad5dd42d5: Time spent bootstrapping tablet: real 0.030s user 0.005s sys 0.007s
I20260925 03:03:54.445966 24662 raft_consensus.cc:348] T 5278a60f97da4e10aecd50a8d9a9a43c P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:54.446661 24662 raft_consensus.cc:374] T 5278a60f97da4e10aecd50a8d9a9a43c P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:54.446923 24662 raft_consensus.cc:729] T 5278a60f97da4e10aecd50a8d9a9a43c P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Initialized, Role: FOLLOWER
I20260925 03:03:54.446966 24664 raft_consensus.cc:348] T a7aa47c48404465b94a36fd819fe3c7a P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:54.447626 24664 raft_consensus.cc:374] T a7aa47c48404465b94a36fd819fe3c7a P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:54.448004 24664 raft_consensus.cc:729] T a7aa47c48404465b94a36fd819fe3c7a P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Initialized, Role: FOLLOWER
I20260925 03:03:54.447654 24662 consensus_queue.cc:260] T 5278a60f97da4e10aecd50a8d9a9a43c P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:54.448607 24664 consensus_queue.cc:260] T a7aa47c48404465b94a36fd819fe3c7a P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:54.452641 24664 ts_tablet_manager.cc:1428] T a7aa47c48404465b94a36fd819fe3c7a P 869b0a8575b54d08ae2817fad5dd42d5: Time spent starting tablet: real 0.010s user 0.003s sys 0.003s
I20260925 03:03:54.453500 24664 tablet_bootstrap.cc:492] T 8ac466793d094325ace224708b2158ed P 869b0a8575b54d08ae2817fad5dd42d5: Bootstrap starting.
W20260925 03:03:54.453806 24588 consensus_peers.cc:487] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c -> Peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187): Couldn't send request to peer 869b0a8575b54d08ae2817fad5dd42d5. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260925 03:03:54.454281 24662 ts_tablet_manager.cc:1428] T 5278a60f97da4e10aecd50a8d9a9a43c P eeab93abfdb141a895002ec9f312296c: Time spent starting tablet: real 0.012s user 0.007s sys 0.000s
I20260925 03:03:54.455188 24662 tablet_bootstrap.cc:492] T 74443a0a1dc34d88a2f42e3b722db405 P eeab93abfdb141a895002ec9f312296c: Bootstrap starting.
I20260925 03:03:54.458791 24664 tablet_bootstrap.cc:654] T 8ac466793d094325ace224708b2158ed P 869b0a8575b54d08ae2817fad5dd42d5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:54.469875 24663 tablet_bootstrap.cc:492] T a4abe0ea757e4886be4945c5001e529a P a04a37fc4aa7460cbfdbeeb4e451e1f5: No bootstrap required, opened a new log
I20260925 03:03:54.470327 24663 ts_tablet_manager.cc:1397] T a4abe0ea757e4886be4945c5001e529a P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent bootstrapping tablet: real 0.046s user 0.016s sys 0.004s
I20260925 03:03:54.472333 24663 raft_consensus.cc:348] T a4abe0ea757e4886be4945c5001e529a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:54.472977 24663 raft_consensus.cc:374] T a4abe0ea757e4886be4945c5001e529a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:54.473229 24663 raft_consensus.cc:729] T a4abe0ea757e4886be4945c5001e529a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Initialized, Role: FOLLOWER
I20260925 03:03:54.473739 24663 consensus_queue.cc:260] T a4abe0ea757e4886be4945c5001e529a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:54.475529 24663 ts_tablet_manager.cc:1428] T a4abe0ea757e4886be4945c5001e529a P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260925 03:03:54.476862 24663 tablet_bootstrap.cc:492] T 5278a60f97da4e10aecd50a8d9a9a43c P a04a37fc4aa7460cbfdbeeb4e451e1f5: Bootstrap starting.
I20260925 03:03:54.478911 24662 tablet_bootstrap.cc:654] T 74443a0a1dc34d88a2f42e3b722db405 P eeab93abfdb141a895002ec9f312296c: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:54.481487 24664 tablet_bootstrap.cc:492] T 8ac466793d094325ace224708b2158ed P 869b0a8575b54d08ae2817fad5dd42d5: No bootstrap required, opened a new log
I20260925 03:03:54.481987 24664 ts_tablet_manager.cc:1397] T 8ac466793d094325ace224708b2158ed P 869b0a8575b54d08ae2817fad5dd42d5: Time spent bootstrapping tablet: real 0.029s user 0.009s sys 0.003s
I20260925 03:03:54.482072 24663 tablet_bootstrap.cc:654] T 5278a60f97da4e10aecd50a8d9a9a43c P a04a37fc4aa7460cbfdbeeb4e451e1f5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:54.484431 24664 raft_consensus.cc:348] T 8ac466793d094325ace224708b2158ed P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:54.485188 24664 raft_consensus.cc:374] T 8ac466793d094325ace224708b2158ed P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:54.485432 24664 raft_consensus.cc:729] T 8ac466793d094325ace224708b2158ed P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Initialized, Role: FOLLOWER
I20260925 03:03:54.486011 24664 consensus_queue.cc:260] T 8ac466793d094325ace224708b2158ed P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:54.492127 24664 ts_tablet_manager.cc:1428] T 8ac466793d094325ace224708b2158ed P 869b0a8575b54d08ae2817fad5dd42d5: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20260925 03:03:54.493059 24664 tablet_bootstrap.cc:492] T a4abe0ea757e4886be4945c5001e529a P 869b0a8575b54d08ae2817fad5dd42d5: Bootstrap starting.
I20260925 03:03:54.500540 24663 tablet_bootstrap.cc:492] T 5278a60f97da4e10aecd50a8d9a9a43c P a04a37fc4aa7460cbfdbeeb4e451e1f5: No bootstrap required, opened a new log
I20260925 03:03:54.501082 24663 ts_tablet_manager.cc:1397] T 5278a60f97da4e10aecd50a8d9a9a43c P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent bootstrapping tablet: real 0.024s user 0.023s sys 0.000s
I20260925 03:03:54.503445 24663 raft_consensus.cc:348] T 5278a60f97da4e10aecd50a8d9a9a43c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:54.504191 24663 raft_consensus.cc:374] T 5278a60f97da4e10aecd50a8d9a9a43c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:54.506283 24664 tablet_bootstrap.cc:654] T a4abe0ea757e4886be4945c5001e529a P 869b0a8575b54d08ae2817fad5dd42d5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:54.504456 24663 raft_consensus.cc:729] T 5278a60f97da4e10aecd50a8d9a9a43c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Initialized, Role: FOLLOWER
I20260925 03:03:54.507352 24663 consensus_queue.cc:260] T 5278a60f97da4e10aecd50a8d9a9a43c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:54.513212 24663 ts_tablet_manager.cc:1428] T 5278a60f97da4e10aecd50a8d9a9a43c P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent starting tablet: real 0.012s user 0.006s sys 0.000s
I20260925 03:03:54.514185 24663 tablet_bootstrap.cc:492] T b030d84c670b48d5bff337ccc0c77626 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Bootstrap starting.
I20260925 03:03:54.514796 24820 raft_consensus.cc:482] T 5278a60f97da4e10aecd50a8d9a9a43c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:54.515308 24820 raft_consensus.cc:504] T 5278a60f97da4e10aecd50a8d9a9a43c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:54.519086 24820 leader_election.cc:288] T 5278a60f97da4e10aecd50a8d9a9a43c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:54.520334 24662 tablet_bootstrap.cc:492] T 74443a0a1dc34d88a2f42e3b722db405 P eeab93abfdb141a895002ec9f312296c: No bootstrap required, opened a new log
I20260925 03:03:54.520206 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5278a60f97da4e10aecd50a8d9a9a43c" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:03:54.520541 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5278a60f97da4e10aecd50a8d9a9a43c" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:03:54.520816 24662 ts_tablet_manager.cc:1397] T 74443a0a1dc34d88a2f42e3b722db405 P eeab93abfdb141a895002ec9f312296c: Time spent bootstrapping tablet: real 0.066s user 0.006s sys 0.008s
I20260925 03:03:54.520866 24663 tablet_bootstrap.cc:654] T b030d84c670b48d5bff337ccc0c77626 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:54.521365 24472 raft_consensus.cc:2454] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 0.
I20260925 03:03:54.521412 24619 raft_consensus.cc:2454] T 5278a60f97da4e10aecd50a8d9a9a43c P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 0.
I20260925 03:03:54.525985 24511 leader_election.cc:302] T 5278a60f97da4e10aecd50a8d9a9a43c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:03:54.526918 24820 raft_consensus.cc:2781] T 5278a60f97da4e10aecd50a8d9a9a43c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 03:03:54.527310 24820 raft_consensus.cc:482] T 5278a60f97da4e10aecd50a8d9a9a43c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:03:54.527626 24820 raft_consensus.cc:3037] T 5278a60f97da4e10aecd50a8d9a9a43c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Advancing to term 1
W20260925 03:03:54.546269 24418 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260925 03:03:54.547179 24664 tablet_bootstrap.cc:492] T a4abe0ea757e4886be4945c5001e529a P 869b0a8575b54d08ae2817fad5dd42d5: No bootstrap required, opened a new log
I20260925 03:03:54.547022 24717 raft_consensus.cc:482] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:54.547649 24664 ts_tablet_manager.cc:1397] T a4abe0ea757e4886be4945c5001e529a P 869b0a8575b54d08ae2817fad5dd42d5: Time spent bootstrapping tablet: real 0.055s user 0.010s sys 0.001s
I20260925 03:03:54.547600 24717 raft_consensus.cc:504] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:54.549823 24717 leader_election.cc:288] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:03:54.550482 24664 raft_consensus.cc:348] T a4abe0ea757e4886be4945c5001e529a P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:54.550637 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e8af2641e124f9589c5ccfdac785f63" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:03:54.551783 24664 raft_consensus.cc:374] T a4abe0ea757e4886be4945c5001e529a P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:54.552099 24664 raft_consensus.cc:729] T a4abe0ea757e4886be4945c5001e529a P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Initialized, Role: FOLLOWER
I20260925 03:03:54.552443 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e8af2641e124f9589c5ccfdac785f63" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:03:54.552986 24663 tablet_bootstrap.cc:492] T b030d84c670b48d5bff337ccc0c77626 P a04a37fc4aa7460cbfdbeeb4e451e1f5: No bootstrap required, opened a new log
I20260925 03:03:54.548890 24820 raft_consensus.cc:504] T 5278a60f97da4e10aecd50a8d9a9a43c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:54.553229 24547 raft_consensus.cc:2454] T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 0.
I20260925 03:03:54.556205 24663 ts_tablet_manager.cc:1397] T b030d84c670b48d5bff337ccc0c77626 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent bootstrapping tablet: real 0.042s user 0.022s sys 0.007s
W20260925 03:03:54.554194 24588 leader_election.cc:341] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:54.553311 24664 consensus_queue.cc:260] T a4abe0ea757e4886be4945c5001e529a P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:54.557973 24586 leader_election.cc:302] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:03:54.559015 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5278a60f97da4e10aecd50a8d9a9a43c" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:03:54.559667 24472 raft_consensus.cc:3037] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:54.559841 24820 leader_election.cc:288] T 5278a60f97da4e10aecd50a8d9a9a43c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:54.561479 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5278a60f97da4e10aecd50a8d9a9a43c" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:03:54.562083 24619 raft_consensus.cc:3037] T 5278a60f97da4e10aecd50a8d9a9a43c P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:54.566277 24472 raft_consensus.cc:2454] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 1.
I20260925 03:03:54.559062 24663 raft_consensus.cc:348] T b030d84c670b48d5bff337ccc0c77626 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:54.567765 24663 raft_consensus.cc:374] T b030d84c670b48d5bff337ccc0c77626 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:54.559074 24717 raft_consensus.cc:2781] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 03:03:54.568166 24663 raft_consensus.cc:729] T b030d84c670b48d5bff337ccc0c77626 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Initialized, Role: FOLLOWER
I20260925 03:03:54.568207 24619 raft_consensus.cc:2454] T 5278a60f97da4e10aecd50a8d9a9a43c P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 1.
I20260925 03:03:54.569062 24514 leader_election.cc:302] T 5278a60f97da4e10aecd50a8d9a9a43c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:03:54.568854 24663 consensus_queue.cc:260] T b030d84c670b48d5bff337ccc0c77626 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:54.569717 24664 ts_tablet_manager.cc:1428] T a4abe0ea757e4886be4945c5001e529a P 869b0a8575b54d08ae2817fad5dd42d5: Time spent starting tablet: real 0.022s user 0.006s sys 0.000s
I20260925 03:03:54.570667 24664 tablet_bootstrap.cc:492] T 9e8af2641e124f9589c5ccfdac785f63 P 869b0a8575b54d08ae2817fad5dd42d5: Bootstrap starting.
I20260925 03:03:54.570920 24770 raft_consensus.cc:2781] T 5278a60f97da4e10aecd50a8d9a9a43c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:03:54.571270 24770 raft_consensus.cc:686] T 5278a60f97da4e10aecd50a8d9a9a43c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 LEADER]: Becoming Leader. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:03:54.571426 24663 ts_tablet_manager.cc:1428] T b030d84c670b48d5bff337ccc0c77626 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent starting tablet: real 0.015s user 0.007s sys 0.000s
I20260925 03:03:54.571734 24717 raft_consensus.cc:482] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:03:54.572105 24717 raft_consensus.cc:3037] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:54.572316 24663 tablet_bootstrap.cc:492] T b4ebd2b043024a548005c8b6b00b0449 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Bootstrap starting.
I20260925 03:03:54.578023 24663 tablet_bootstrap.cc:654] T b4ebd2b043024a548005c8b6b00b0449 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:54.578889 24770 consensus_queue.cc:237] T 5278a60f97da4e10aecd50a8d9a9a43c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:54.579931 24664 tablet_bootstrap.cc:654] T 9e8af2641e124f9589c5ccfdac785f63 P 869b0a8575b54d08ae2817fad5dd42d5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:54.545492 24662 raft_consensus.cc:348] T 74443a0a1dc34d88a2f42e3b722db405 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:54.581588 24662 raft_consensus.cc:374] T 74443a0a1dc34d88a2f42e3b722db405 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:54.581874 24662 raft_consensus.cc:729] T 74443a0a1dc34d88a2f42e3b722db405 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Initialized, Role: FOLLOWER
I20260925 03:03:54.582509 24662 consensus_queue.cc:260] T 74443a0a1dc34d88a2f42e3b722db405 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:54.578169 24622 raft_consensus.cc:3037] T c85f5d1d3f004cf5817bb594fd8e6469 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:54.584846 24662 ts_tablet_manager.cc:1428] T 74443a0a1dc34d88a2f42e3b722db405 P eeab93abfdb141a895002ec9f312296c: Time spent starting tablet: real 0.064s user 0.013s sys 0.001s
I20260925 03:03:54.585913 24662 tablet_bootstrap.cc:492] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P eeab93abfdb141a895002ec9f312296c: Bootstrap starting.
I20260925 03:03:54.588922 24663 tablet_bootstrap.cc:492] T b4ebd2b043024a548005c8b6b00b0449 P a04a37fc4aa7460cbfdbeeb4e451e1f5: No bootstrap required, opened a new log
I20260925 03:03:54.589395 24663 ts_tablet_manager.cc:1397] T b4ebd2b043024a548005c8b6b00b0449 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent bootstrapping tablet: real 0.017s user 0.012s sys 0.001s
I20260925 03:03:54.596673 24663 raft_consensus.cc:348] T b4ebd2b043024a548005c8b6b00b0449 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:54.597473 24663 raft_consensus.cc:374] T b4ebd2b043024a548005c8b6b00b0449 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:54.597718 24663 raft_consensus.cc:729] T b4ebd2b043024a548005c8b6b00b0449 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Initialized, Role: FOLLOWER
I20260925 03:03:54.598296 24663 consensus_queue.cc:260] T b4ebd2b043024a548005c8b6b00b0449 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:54.600118 24663 ts_tablet_manager.cc:1428] T b4ebd2b043024a548005c8b6b00b0449 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent starting tablet: real 0.010s user 0.003s sys 0.001s
I20260925 03:03:54.602520 24663 tablet_bootstrap.cc:492] T 37a157996e944db2906110d87ecb0c54 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Bootstrap starting.
I20260925 03:03:54.602391 24771 raft_consensus.cc:482] T 8ac466793d094325ace224708b2158ed P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:54.603099 24771 raft_consensus.cc:504] T 8ac466793d094325ace224708b2158ed P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:54.613364 24771 leader_election.cc:288] T 8ac466793d094325ace224708b2158ed P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:54.614140 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ac466793d094325ace224708b2158ed" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:03:54.600144 24717 raft_consensus.cc:504] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:54.623888 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ac466793d094325ace224708b2158ed" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:03:54.626156 24437 leader_election.cc:302] T 8ac466793d094325ace224708b2158ed P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:03:54.628358 24771 raft_consensus.cc:2726] T 8ac466793d094325ace224708b2158ed P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 03:03:54.624293 24717 leader_election.cc:288] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 1 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:03:54.630167 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e8af2641e124f9589c5ccfdac785f63" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:03:54.632138 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e8af2641e124f9589c5ccfdac785f63" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
W20260925 03:03:54.632707 24588 leader_election.cc:341] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:54.634135 24547 raft_consensus.cc:3037] T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:54.639074 24814 consensus_queue.cc:1035] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260925 03:03:54.643711 24368 catalog_manager.cc:5495] T 5278a60f97da4e10aecd50a8d9a9a43c P a04a37fc4aa7460cbfdbeeb4e451e1f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2). New cstate: current_term: 1 leader_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: UNKNOWN } } }
I20260925 03:03:54.652767 24547 raft_consensus.cc:2454] T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 1.
I20260925 03:03:54.653963 24586 leader_election.cc:302] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:03:54.658711 24717 raft_consensus.cc:2781] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:03:54.659273 24717 raft_consensus.cc:686] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c [term 1 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:03:54.660245 24664 tablet_bootstrap.cc:492] T 9e8af2641e124f9589c5ccfdac785f63 P 869b0a8575b54d08ae2817fad5dd42d5: No bootstrap required, opened a new log
I20260925 03:03:54.661252 24717 consensus_queue.cc:237] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:54.660651 24664 ts_tablet_manager.cc:1397] T 9e8af2641e124f9589c5ccfdac785f63 P 869b0a8575b54d08ae2817fad5dd42d5: Time spent bootstrapping tablet: real 0.090s user 0.013s sys 0.020s
I20260925 03:03:54.666766 24814 consensus_queue.cc:1035] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20260925 03:03:54.679395 24708 raft_consensus.cc:482] T a7aa47c48404465b94a36fd819fe3c7a P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:54.679976 24708 raft_consensus.cc:504] T a7aa47c48404465b94a36fd819fe3c7a P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:54.686352 24664 raft_consensus.cc:348] T 9e8af2641e124f9589c5ccfdac785f63 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:54.687119 24664 raft_consensus.cc:374] T 9e8af2641e124f9589c5ccfdac785f63 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:54.687376 24664 raft_consensus.cc:729] T 9e8af2641e124f9589c5ccfdac785f63 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Initialized, Role: FOLLOWER
I20260925 03:03:54.688004 24664 consensus_queue.cc:260] T 9e8af2641e124f9589c5ccfdac785f63 P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:54.691287 24718 raft_consensus.cc:482] T 43366a0a2f9d4344bf4e37ffd290ef8a P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:54.692952 24718 raft_consensus.cc:504] T 43366a0a2f9d4344bf4e37ffd290ef8a P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:54.691902 24814 raft_consensus.cc:482] T a4abe0ea757e4886be4945c5001e529a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:54.694077 24664 ts_tablet_manager.cc:1428] T 9e8af2641e124f9589c5ccfdac785f63 P 869b0a8575b54d08ae2817fad5dd42d5: Time spent starting tablet: real 0.031s user 0.006s sys 0.000s
I20260925 03:03:54.694250 24814 raft_consensus.cc:504] T a4abe0ea757e4886be4945c5001e529a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:54.695010 24664 tablet_bootstrap.cc:492] T 3d89396576884f7c8901dba54f783947 P 869b0a8575b54d08ae2817fad5dd42d5: Bootstrap starting.
I20260925 03:03:54.696166 24814 leader_election.cc:288] T a4abe0ea757e4886be4945c5001e529a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:54.698237 24708 leader_election.cc:288] T a7aa47c48404465b94a36fd819fe3c7a P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:54.699724 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7aa47c48404465b94a36fd819fe3c7a" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:03:54.700387 24546 raft_consensus.cc:2454] T a7aa47c48404465b94a36fd819fe3c7a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 0.
I20260925 03:03:54.701239 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4abe0ea757e4886be4945c5001e529a" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:03:54.701573 24770 raft_consensus.cc:482] T b4ebd2b043024a548005c8b6b00b0449 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:54.701949 24473 raft_consensus.cc:2454] T a4abe0ea757e4886be4945c5001e529a P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 0.
I20260925 03:03:54.702011 24770 raft_consensus.cc:504] T b4ebd2b043024a548005c8b6b00b0449 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:54.694943 24718 leader_election.cc:288] T 43366a0a2f9d4344bf4e37ffd290ef8a P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:03:54.707249 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7aa47c48404465b94a36fd819fe3c7a" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:03:54.708549 24770 leader_election.cc:288] T b4ebd2b043024a548005c8b6b00b0449 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:54.709425 24708 raft_consensus.cc:482] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:54.712025 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43366a0a2f9d4344bf4e37ffd290ef8a" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:03:54.718467 24708 raft_consensus.cc:504] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:54.721226 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43366a0a2f9d4344bf4e37ffd290ef8a" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:03:54.722159 24546 raft_consensus.cc:2379] T 43366a0a2f9d4344bf4e37ffd290ef8a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c in current term 1: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:03:54.723848 24586 leader_election.cc:302] T 43366a0a2f9d4344bf4e37ffd290ef8a P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:03:54.712533 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4abe0ea757e4886be4945c5001e529a" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:03:54.725895 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b030d84c670b48d5bff337ccc0c77626" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:03:54.731797 24718 raft_consensus.cc:3037] T 43366a0a2f9d4344bf4e37ffd290ef8a P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Advancing to term 1
W20260925 03:03:54.742632 24436 leader_election.cc:341] T a7aa47c48404465b94a36fd819fe3c7a P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105): Illegal state: must be running to vote when last-logged opid is not known
W20260925 03:03:54.756270 24436 leader_election.cc:341] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:54.758165 24662 tablet_bootstrap.cc:654] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P eeab93abfdb141a895002ec9f312296c: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:54.732319 24708 leader_election.cc:288] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:54.759359 24663 tablet_bootstrap.cc:654] T 37a157996e944db2906110d87ecb0c54 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:54.760665 24437 leader_election.cc:302] T a7aa47c48404465b94a36fd819fe3c7a P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
W20260925 03:03:54.762907 24511 leader_election.cc:341] T a4abe0ea757e4886be4945c5001e529a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:54.764366 24770 raft_consensus.cc:482] T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:54.766083 24671 raft_consensus.cc:2781] T a7aa47c48404465b94a36fd819fe3c7a P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 03:03:54.766451 24671 raft_consensus.cc:482] T a7aa47c48404465b94a36fd819fe3c7a P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:03:54.766750 24671 raft_consensus.cc:3037] T a7aa47c48404465b94a36fd819fe3c7a P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:54.767596 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4ebd2b043024a548005c8b6b00b0449" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:03:54.768071 24718 raft_consensus.cc:2726] T 43366a0a2f9d4344bf4e37ffd290ef8a P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 03:03:54.760273 24371 catalog_manager.cc:5495] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c reported cstate change: term changed from 0 to 1, leader changed from <none> to eeab93abfdb141a895002ec9f312296c (127.22.193.3). New cstate: current_term: 1 leader_uuid: "eeab93abfdb141a895002ec9f312296c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: HEALTHY } } }
I20260925 03:03:54.769037 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4ebd2b043024a548005c8b6b00b0449" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:03:54.773233 24470 raft_consensus.cc:2454] T b4ebd2b043024a548005c8b6b00b0449 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 0.
I20260925 03:03:54.769672 24770 raft_consensus.cc:504] T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:54.769956 24717 raft_consensus.cc:482] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:54.778939 24514 leader_election.cc:302] T a4abe0ea757e4886be4945c5001e529a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:03:54.782393 24763 raft_consensus.cc:2781] T a4abe0ea757e4886be4945c5001e529a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 03:03:54.782800 24763 raft_consensus.cc:482] T a4abe0ea757e4886be4945c5001e529a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:03:54.783129 24763 raft_consensus.cc:3037] T a4abe0ea757e4886be4945c5001e529a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:54.787707 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b030d84c670b48d5bff337ccc0c77626" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:03:54.791481 24770 leader_election.cc:288] T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:54.811765 24664 tablet_bootstrap.cc:654] T 3d89396576884f7c8901dba54f783947 P 869b0a8575b54d08ae2817fad5dd42d5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:54.831923 24763 raft_consensus.cc:504] T a4abe0ea757e4886be4945c5001e529a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:03:54.779858 24511 leader_election.cc:341] T b4ebd2b043024a548005c8b6b00b0449 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105): Illegal state: must be running to vote when last-logged opid is not known
W20260925 03:03:54.838570 24511 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 58770.2us
I20260925 03:03:54.839627 24763 leader_election.cc:288] T a4abe0ea757e4886be4945c5001e529a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:54.804615 24671 raft_consensus.cc:504] T a7aa47c48404465b94a36fd819fe3c7a P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:54.780014 24717 raft_consensus.cc:504] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:54.813862 24708 consensus_queue.cc:1035] T 43366a0a2f9d4344bf4e37ffd290ef8a P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.018s
I20260925 03:03:54.809978 24546 raft_consensus.cc:2454] T b030d84c670b48d5bff337ccc0c77626 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 0.
I20260925 03:03:54.845450 24663 tablet_bootstrap.cc:492] T 37a157996e944db2906110d87ecb0c54 P a04a37fc4aa7460cbfdbeeb4e451e1f5: No bootstrap required, opened a new log
I20260925 03:03:54.848280 24620 raft_consensus.cc:3037] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:54.848672 24663 ts_tablet_manager.cc:1397] T 37a157996e944db2906110d87ecb0c54 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent bootstrapping tablet: real 0.246s user 0.026s sys 0.024s
W20260925 03:03:54.846526 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 74551.7us
I20260925 03:03:54.847395 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5332a5d329493991dff994e70ea92c" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:03:54.848946 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5332a5d329493991dff994e70ea92c" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:03:54.846030 24717 leader_election.cc:288] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:03:54.851802 24588 leader_election.cc:302] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:03:54.853812 24663 raft_consensus.cc:348] T 37a157996e944db2906110d87ecb0c54 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:54.854547 24663 raft_consensus.cc:374] T 37a157996e944db2906110d87ecb0c54 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:54.854804 24663 raft_consensus.cc:729] T 37a157996e944db2906110d87ecb0c54 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Initialized, Role: FOLLOWER
I20260925 03:03:54.855427 24663 consensus_queue.cc:260] T 37a157996e944db2906110d87ecb0c54 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:54.858289 24514 leader_election.cc:302] T b4ebd2b043024a548005c8b6b00b0449 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:03:54.866369 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74443a0a1dc34d88a2f42e3b722db405" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:03:54.866926 24617 raft_consensus.cc:2454] T 74443a0a1dc34d88a2f42e3b722db405 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 0.
I20260925 03:03:54.857815 24671 leader_election.cc:288] T a7aa47c48404465b94a36fd819fe3c7a P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:54.873551 24663 ts_tablet_manager.cc:1428] T 37a157996e944db2906110d87ecb0c54 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent starting tablet: real 0.022s user 0.005s sys 0.001s
I20260925 03:03:54.873764 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7aa47c48404465b94a36fd819fe3c7a" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:03:54.879091 24548 raft_consensus.cc:3037] T a7aa47c48404465b94a36fd819fe3c7a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:54.879557 24663 tablet_bootstrap.cc:492] T 3d89396576884f7c8901dba54f783947 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Bootstrap starting.
I20260925 03:03:54.880273 24671 consensus_queue.cc:1035] T 43366a0a2f9d4344bf4e37ffd290ef8a P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260925 03:03:54.875061 24770 raft_consensus.cc:2781] T b4ebd2b043024a548005c8b6b00b0449 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 03:03:54.885064 24770 raft_consensus.cc:482] T b4ebd2b043024a548005c8b6b00b0449 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:03:54.885529 24770 raft_consensus.cc:3037] T b4ebd2b043024a548005c8b6b00b0449 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:54.886338 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74443a0a1dc34d88a2f42e3b722db405" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:03:54.886966 24470 raft_consensus.cc:2454] T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 0.
I20260925 03:03:54.877394 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7aa47c48404465b94a36fd819fe3c7a" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
W20260925 03:03:54.888200 24436 leader_election.cc:341] T a7aa47c48404465b94a36fd819fe3c7a P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:54.889146 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4abe0ea757e4886be4945c5001e529a" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:03:54.889842 24470 raft_consensus.cc:3037] T a4abe0ea757e4886be4945c5001e529a P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:54.889979 24511 leader_election.cc:302] T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:03:54.892763 24763 raft_consensus.cc:2781] T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 03:03:54.893177 24763 raft_consensus.cc:482] T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:03:54.893616 24763 raft_consensus.cc:3037] T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:54.892939 24548 raft_consensus.cc:2454] T a7aa47c48404465b94a36fd819fe3c7a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 1.
I20260925 03:03:54.896944 24664 tablet_bootstrap.cc:492] T 3d89396576884f7c8901dba54f783947 P 869b0a8575b54d08ae2817fad5dd42d5: No bootstrap required, opened a new log
I20260925 03:03:54.899883 24662 tablet_bootstrap.cc:492] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P eeab93abfdb141a895002ec9f312296c: No bootstrap required, opened a new log
I20260925 03:03:54.901444 24664 ts_tablet_manager.cc:1397] T 3d89396576884f7c8901dba54f783947 P 869b0a8575b54d08ae2817fad5dd42d5: Time spent bootstrapping tablet: real 0.207s user 0.040s sys 0.011s
I20260925 03:03:54.901705 24662 ts_tablet_manager.cc:1397] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P eeab93abfdb141a895002ec9f312296c: Time spent bootstrapping tablet: real 0.316s user 0.015s sys 0.017s
I20260925 03:03:54.900493 24470 raft_consensus.cc:2454] T a4abe0ea757e4886be4945c5001e529a P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 1.
I20260925 03:03:54.907627 24814 raft_consensus.cc:482] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:54.916224 24814 raft_consensus.cc:504] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:54.916978 24437 leader_election.cc:302] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:03:54.917266 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4abe0ea757e4886be4945c5001e529a" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:03:54.918601 24814 leader_election.cc:288] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:54.918876 24662 raft_consensus.cc:348] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:54.919610 24662 raft_consensus.cc:374] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:54.919922 24662 raft_consensus.cc:729] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Initialized, Role: FOLLOWER
I20260925 03:03:54.920506 24662 consensus_queue.cc:260] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:54.918916 24871 raft_consensus.cc:2781] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 03:03:54.922806 24871 raft_consensus.cc:482] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:03:54.923125 24871 raft_consensus.cc:3037] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:54.926245 24763 raft_consensus.cc:504] T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:54.926618 24437 leader_election.cc:302] T a7aa47c48404465b94a36fd819fe3c7a P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:03:54.928799 24663 tablet_bootstrap.cc:654] T 3d89396576884f7c8901dba54f783947 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:54.937181 24662 ts_tablet_manager.cc:1428] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P eeab93abfdb141a895002ec9f312296c: Time spent starting tablet: real 0.035s user 0.005s sys 0.001s
I20260925 03:03:54.939535 24662 tablet_bootstrap.cc:492] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c: Bootstrap starting.
I20260925 03:03:54.933575 24717 raft_consensus.cc:2726] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 03:03:54.940857 24664 raft_consensus.cc:348] T 3d89396576884f7c8901dba54f783947 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:54.941833 24664 raft_consensus.cc:374] T 3d89396576884f7c8901dba54f783947 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:54.942469 24664 raft_consensus.cc:729] T 3d89396576884f7c8901dba54f783947 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Initialized, Role: FOLLOWER
I20260925 03:03:54.943074 24664 consensus_queue.cc:260] T 3d89396576884f7c8901dba54f783947 P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:54.941810 24708 raft_consensus.cc:2781] T a7aa47c48404465b94a36fd819fe3c7a P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:03:54.942766 24470 raft_consensus.cc:3037] T 8ac466793d094325ace224708b2158ed P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:54.952942 24664 ts_tablet_manager.cc:1428] T 3d89396576884f7c8901dba54f783947 P 869b0a8575b54d08ae2817fad5dd42d5: Time spent starting tablet: real 0.051s user 0.004s sys 0.002s
I20260925 03:03:54.953902 24664 tablet_bootstrap.cc:492] T 37a157996e944db2906110d87ecb0c54 P 869b0a8575b54d08ae2817fad5dd42d5: Bootstrap starting.
I20260925 03:03:54.937220 24770 raft_consensus.cc:504] T b4ebd2b043024a548005c8b6b00b0449 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:54.971699 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abd5b8dc1b7e49e3aa9af900d73ff0e7" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:03:54.972487 24620 raft_consensus.cc:2454] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 0.
W20260925 03:03:54.972532 24511 leader_election.cc:341] T a4abe0ea757e4886be4945c5001e529a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:54.973249 24511 leader_election.cc:302] T a4abe0ea757e4886be4945c5001e529a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:03:54.974261 24662 tablet_bootstrap.cc:654] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:54.974602 24814 raft_consensus.cc:2781] T a4abe0ea757e4886be4945c5001e529a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:03:54.962745 24708 raft_consensus.cc:686] T a7aa47c48404465b94a36fd819fe3c7a P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 LEADER]: Becoming Leader. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:03:54.975087 24814 raft_consensus.cc:686] T a4abe0ea757e4886be4945c5001e529a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 LEADER]: Becoming Leader. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:03:54.977880 24708 consensus_queue.cc:237] T a7aa47c48404465b94a36fd819fe3c7a P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:54.978044 24871 raft_consensus.cc:504] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:54.977969 24814 consensus_queue.cc:237] T a4abe0ea757e4886be4945c5001e529a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:54.979655 24770 leader_election.cc:288] T b4ebd2b043024a548005c8b6b00b0449 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:54.980537 24871 leader_election.cc:288] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:54.969485 24763 leader_election.cc:288] T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:54.976372 24511 leader_election.cc:302] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:03:54.986002 24763 raft_consensus.cc:2781] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 03:03:54.986131 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b030d84c670b48d5bff337ccc0c77626" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:03:54.986387 24763 raft_consensus.cc:482] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:03:54.986773 24763 raft_consensus.cc:3037] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:54.991600 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b030d84c670b48d5bff337ccc0c77626" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:03:54.992236 24548 raft_consensus.cc:3037] T b030d84c670b48d5bff337ccc0c77626 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:54.993039 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74443a0a1dc34d88a2f42e3b722db405" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:03:54.993603 24622 raft_consensus.cc:3037] T 74443a0a1dc34d88a2f42e3b722db405 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:55.017907 24880 raft_consensus.cc:482] T 3d89396576884f7c8901dba54f783947 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:55.019980 24663 tablet_bootstrap.cc:492] T 3d89396576884f7c8901dba54f783947 P a04a37fc4aa7460cbfdbeeb4e451e1f5: No bootstrap required, opened a new log
I20260925 03:03:55.020429 24663 ts_tablet_manager.cc:1397] T 3d89396576884f7c8901dba54f783947 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent bootstrapping tablet: real 0.141s user 0.009s sys 0.011s
W20260925 03:03:54.998313 24436 leader_election.cc:341] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:55.021224 24880 raft_consensus.cc:504] T 3d89396576884f7c8901dba54f783947 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:55.027073 24662 tablet_bootstrap.cc:492] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c: No bootstrap required, opened a new log
I20260925 03:03:55.027596 24662 ts_tablet_manager.cc:1397] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c: Time spent bootstrapping tablet: real 0.088s user 0.013s sys 0.004s
I20260925 03:03:54.999750 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4ebd2b043024a548005c8b6b00b0449" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:03:55.015893 24622 raft_consensus.cc:2454] T 74443a0a1dc34d88a2f42e3b722db405 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 1.
I20260925 03:03:55.008376 24548 raft_consensus.cc:2454] T b030d84c670b48d5bff337ccc0c77626 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 1.
I20260925 03:03:55.030591 24820 raft_consensus.cc:482] T 37a157996e944db2906110d87ecb0c54 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:55.032336 24820 raft_consensus.cc:504] T 37a157996e944db2906110d87ecb0c54 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:55.033844 24880 leader_election.cc:288] T 3d89396576884f7c8901dba54f783947 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:55.035118 24820 leader_election.cc:288] T 37a157996e944db2906110d87ecb0c54 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:55.035247 24663 raft_consensus.cc:348] T 3d89396576884f7c8901dba54f783947 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:55.039547 24663 raft_consensus.cc:374] T 3d89396576884f7c8901dba54f783947 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:55.039954 24663 raft_consensus.cc:729] T 3d89396576884f7c8901dba54f783947 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Initialized, Role: FOLLOWER
I20260925 03:03:55.039570 24662 raft_consensus.cc:348] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:55.040918 24662 raft_consensus.cc:374] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:55.041263 24662 raft_consensus.cc:729] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Initialized, Role: FOLLOWER
I20260925 03:03:55.042503 24663 consensus_queue.cc:260] T 3d89396576884f7c8901dba54f783947 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:55.042058 24662 consensus_queue.cc:260] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:55.045486 24662 ts_tablet_manager.cc:1428] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c: Time spent starting tablet: real 0.018s user 0.006s sys 0.001s
I20260925 03:03:55.046792 24662 tablet_bootstrap.cc:492] T b4ebd2b043024a548005c8b6b00b0449 P eeab93abfdb141a895002ec9f312296c: Bootstrap starting.
I20260925 03:03:55.039186 24664 tablet_bootstrap.cc:654] T 37a157996e944db2906110d87ecb0c54 P 869b0a8575b54d08ae2817fad5dd42d5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:55.049724 24371 catalog_manager.cc:5495] T a4abe0ea757e4886be4945c5001e529a P a04a37fc4aa7460cbfdbeeb4e451e1f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2). New cstate: current_term: 1 leader_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: UNKNOWN } } }
I20260925 03:03:55.057327 24663 ts_tablet_manager.cc:1428] T 3d89396576884f7c8901dba54f783947 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent starting tablet: real 0.037s user 0.007s sys 0.000s
I20260925 03:03:55.057190 24511 leader_election.cc:302] T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:03:55.058517 24663 tablet_bootstrap.cc:492] T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Bootstrap starting.
I20260925 03:03:55.058983 24820 raft_consensus.cc:2781] T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:03:55.059603 24820 raft_consensus.cc:686] T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 LEADER]: Becoming Leader. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:03:55.060809 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d89396576884f7c8901dba54f783947" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:03:55.060885 24820 consensus_queue.cc:237] T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:55.053133 24368 catalog_manager.cc:5495] T a7aa47c48404465b94a36fd819fe3c7a P 869b0a8575b54d08ae2817fad5dd42d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1). New cstate: current_term: 1 leader_uuid: "869b0a8575b54d08ae2817fad5dd42d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: UNKNOWN } } }
W20260925 03:03:55.058734 24511 leader_election.cc:341] T b4ebd2b043024a548005c8b6b00b0449 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:55.076354 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abd5b8dc1b7e49e3aa9af900d73ff0e7" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:03:55.077121 24472 raft_consensus.cc:2454] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 0.
I20260925 03:03:55.072244 24880 raft_consensus.cc:482] T 9e8af2641e124f9589c5ccfdac785f63 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:55.078778 24880 raft_consensus.cc:504] T 9e8af2641e124f9589c5ccfdac785f63 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:55.083153 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37a157996e944db2906110d87ecb0c54" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
W20260925 03:03:55.074252 24436 leader_election.cc:341] T 3d89396576884f7c8901dba54f783947 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:55.087502 24880 leader_election.cc:288] T 9e8af2641e124f9589c5ccfdac785f63 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:55.088912 24662 tablet_bootstrap.cc:654] T b4ebd2b043024a548005c8b6b00b0449 P eeab93abfdb141a895002ec9f312296c: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:55.094040 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e8af2641e124f9589c5ccfdac785f63" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:03:55.099702 24663 tablet_bootstrap.cc:654] T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:55.094134 24763 raft_consensus.cc:504] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:55.102891 24763 leader_election.cc:288] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:55.106006 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74443a0a1dc34d88a2f42e3b722db405" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:03:55.107221 24473 raft_consensus.cc:3037] T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:55.108203 24664 tablet_bootstrap.cc:492] T 37a157996e944db2906110d87ecb0c54 P 869b0a8575b54d08ae2817fad5dd42d5: No bootstrap required, opened a new log
I20260925 03:03:55.108635 24664 ts_tablet_manager.cc:1397] T 37a157996e944db2906110d87ecb0c54 P 869b0a8575b54d08ae2817fad5dd42d5: Time spent bootstrapping tablet: real 0.155s user 0.013s sys 0.008s
I20260925 03:03:55.106391 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4ebd2b043024a548005c8b6b00b0449" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:03:55.110250 24472 raft_consensus.cc:3037] T b4ebd2b043024a548005c8b6b00b0449 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Advancing to term 1
W20260925 03:03:55.110033 24511 leader_election.cc:341] T 37a157996e944db2906110d87ecb0c54 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:55.111529 24664 raft_consensus.cc:348] T 37a157996e944db2906110d87ecb0c54 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:55.112243 24664 raft_consensus.cc:374] T 37a157996e944db2906110d87ecb0c54 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:55.112541 24664 raft_consensus.cc:729] T 37a157996e944db2906110d87ecb0c54 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Initialized, Role: FOLLOWER
I20260925 03:03:55.113971 24664 consensus_queue.cc:260] T 37a157996e944db2906110d87ecb0c54 P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:55.115402 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abd5b8dc1b7e49e3aa9af900d73ff0e7" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:03:55.116030 24622 raft_consensus.cc:3037] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:55.117511 24664 ts_tablet_manager.cc:1428] T 37a157996e944db2906110d87ecb0c54 P 869b0a8575b54d08ae2817fad5dd42d5: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20260925 03:03:55.118479 24664 tablet_bootstrap.cc:492] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5: Bootstrap starting.
I20260925 03:03:55.120936 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37a157996e944db2906110d87ecb0c54" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:03:55.121592 24471 raft_consensus.cc:2454] T 37a157996e944db2906110d87ecb0c54 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 0.
I20260925 03:03:55.140297 24473 raft_consensus.cc:2454] T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 1.
I20260925 03:03:55.138146 24663 tablet_bootstrap.cc:492] T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5: No bootstrap required, opened a new log
I20260925 03:03:55.141343 24663 ts_tablet_manager.cc:1397] T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent bootstrapping tablet: real 0.083s user 0.020s sys 0.012s
I20260925 03:03:55.142156 24472 raft_consensus.cc:2454] T b4ebd2b043024a548005c8b6b00b0449 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 1.
I20260925 03:03:55.143509 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abd5b8dc1b7e49e3aa9af900d73ff0e7" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:03:55.143314 24368 catalog_manager.cc:5495] T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2). New cstate: current_term: 1 leader_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: UNKNOWN } } }
I20260925 03:03:55.144121 24471 raft_consensus.cc:3037] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:55.144912 24820 consensus_queue.cc:1035] T 5278a60f97da4e10aecd50a8d9a9a43c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 03:03:55.146334 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d89396576884f7c8901dba54f783947" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:03:55.147002 24547 raft_consensus.cc:2454] T 3d89396576884f7c8901dba54f783947 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 0.
I20260925 03:03:55.161625 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e8af2641e124f9589c5ccfdac785f63" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:03:55.162462 24548 raft_consensus.cc:2379] T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 in current term 1: Already voted for candidate eeab93abfdb141a895002ec9f312296c in this term.
I20260925 03:03:55.162460 24622 raft_consensus.cc:2454] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 1.
I20260925 03:03:55.163172 24514 leader_election.cc:302] T 37a157996e944db2906110d87ecb0c54 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:03:55.163884 24511 leader_election.cc:302] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:03:55.164877 24814 raft_consensus.cc:2781] T 37a157996e944db2906110d87ecb0c54 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 03:03:55.165618 24814 raft_consensus.cc:482] T 37a157996e944db2906110d87ecb0c54 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:03:55.168612 24814 raft_consensus.cc:3037] T 37a157996e944db2906110d87ecb0c54 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:55.169447 24514 leader_election.cc:302] T b4ebd2b043024a548005c8b6b00b0449 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:03:55.170221 24763 raft_consensus.cc:2781] T b4ebd2b043024a548005c8b6b00b0449 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:03:55.170698 24763 raft_consensus.cc:686] T b4ebd2b043024a548005c8b6b00b0449 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 LEADER]: Becoming Leader. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:03:55.171478 24763 consensus_queue.cc:237] T b4ebd2b043024a548005c8b6b00b0449 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:55.166115 24820 raft_consensus.cc:2781] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:03:55.174857 24820 raft_consensus.cc:686] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 LEADER]: Becoming Leader. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:03:55.165213 24471 raft_consensus.cc:2454] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 1.
I20260925 03:03:55.177573 24664 tablet_bootstrap.cc:654] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:55.175595 24814 raft_consensus.cc:504] T 37a157996e944db2906110d87ecb0c54 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:55.185305 24814 leader_election.cc:288] T 37a157996e944db2906110d87ecb0c54 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:55.175730 24820 consensus_queue.cc:237] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:55.194756 24662 tablet_bootstrap.cc:492] T b4ebd2b043024a548005c8b6b00b0449 P eeab93abfdb141a895002ec9f312296c: No bootstrap required, opened a new log
I20260925 03:03:55.195252 24662 ts_tablet_manager.cc:1397] T b4ebd2b043024a548005c8b6b00b0449 P eeab93abfdb141a895002ec9f312296c: Time spent bootstrapping tablet: real 0.149s user 0.012s sys 0.006s
I20260925 03:03:55.195405 24663 raft_consensus.cc:348] T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:55.196300 24663 raft_consensus.cc:374] T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:55.196595 24663 raft_consensus.cc:729] T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Initialized, Role: FOLLOWER
I20260925 03:03:55.197280 24663 consensus_queue.cc:260] T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:55.197621 24662 raft_consensus.cc:348] T b4ebd2b043024a548005c8b6b00b0449 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:55.199905 24662 raft_consensus.cc:374] T b4ebd2b043024a548005c8b6b00b0449 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:55.200122 24662 raft_consensus.cc:729] T b4ebd2b043024a548005c8b6b00b0449 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Initialized, Role: FOLLOWER
I20260925 03:03:55.202564 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37a157996e944db2906110d87ecb0c54" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
W20260925 03:03:55.203859 24511 leader_election.cc:341] T 37a157996e944db2906110d87ecb0c54 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:55.205029 24437 leader_election.cc:302] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:03:55.204720 24662 consensus_queue.cc:260] T b4ebd2b043024a548005c8b6b00b0449 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:55.211076 24880 raft_consensus.cc:2781] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:03:55.211707 24880 raft_consensus.cc:686] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 LEADER]: Becoming Leader. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:03:55.212927 24880 consensus_queue.cc:237] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:55.218851 24437 leader_election.cc:302] T 3d89396576884f7c8901dba54f783947 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:03:55.219766 24885 raft_consensus.cc:2781] T 3d89396576884f7c8901dba54f783947 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 03:03:55.220136 24885 raft_consensus.cc:482] T 3d89396576884f7c8901dba54f783947 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:03:55.220458 24885 raft_consensus.cc:3037] T 3d89396576884f7c8901dba54f783947 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:55.220458 24368 catalog_manager.cc:5495] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P a04a37fc4aa7460cbfdbeeb4e451e1f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2). New cstate: current_term: 1 leader_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: UNKNOWN } } }
I20260925 03:03:55.222597 24368 catalog_manager.cc:5495] T b4ebd2b043024a548005c8b6b00b0449 P a04a37fc4aa7460cbfdbeeb4e451e1f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2). New cstate: current_term: 1 leader_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: UNKNOWN } } }
I20260925 03:03:55.229758 24662 ts_tablet_manager.cc:1428] T b4ebd2b043024a548005c8b6b00b0449 P eeab93abfdb141a895002ec9f312296c: Time spent starting tablet: real 0.034s user 0.006s sys 0.001s
I20260925 03:03:55.225435 24437 leader_election.cc:302] T 9e8af2641e124f9589c5ccfdac785f63 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:03:55.233740 24879 raft_consensus.cc:3037] T 9e8af2641e124f9589c5ccfdac785f63 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:55.234263 24670 consensus_queue.cc:1035] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 03:03:55.235937 24662 tablet_bootstrap.cc:492] T a4abe0ea757e4886be4945c5001e529a P eeab93abfdb141a895002ec9f312296c: Bootstrap starting.
I20260925 03:03:55.206943 24663 ts_tablet_manager.cc:1428] T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent starting tablet: real 0.065s user 0.004s sys 0.002s
I20260925 03:03:55.238363 24670 raft_consensus.cc:482] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:55.238826 24670 raft_consensus.cc:504] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:55.239053 24820 consensus_queue.cc:1035] T 5278a60f97da4e10aecd50a8d9a9a43c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 03:03:55.241329 24670 leader_election.cc:288] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:03:55.241622 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b030d84c670b48d5bff337ccc0c77626" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:03:55.243098 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b030d84c670b48d5bff337ccc0c77626" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:03:55.244939 24546 raft_consensus.cc:2379] T b030d84c670b48d5bff337ccc0c77626 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c in current term 1: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:03:55.246181 24663 tablet_bootstrap.cc:492] T 1414dcfe53f64a78ae413531eb8ec117 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Bootstrap starting.
I20260925 03:03:55.246362 24879 raft_consensus.cc:2726] T 9e8af2641e124f9589c5ccfdac785f63 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 03:03:55.247972 24820 raft_consensus.cc:482] T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:55.248456 24820 raft_consensus.cc:504] T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:55.249982 24586 leader_election.cc:302] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:03:55.250727 24670 raft_consensus.cc:3037] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:55.251281 24820 leader_election.cc:288] T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:55.251504 24664 tablet_bootstrap.cc:492] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5: No bootstrap required, opened a new log
I20260925 03:03:55.251986 24664 ts_tablet_manager.cc:1397] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5: Time spent bootstrapping tablet: real 0.134s user 0.012s sys 0.026s
I20260925 03:03:55.254390 24717 consensus_queue.cc:1035] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 03:03:55.257926 24670 raft_consensus.cc:2726] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 03:03:55.259677 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c23337a7cb74184a0e57b7095cb6a11" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
W20260925 03:03:55.261981 24511 leader_election.cc:341] T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:55.266495 24664 raft_consensus.cc:348] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:55.266937 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37a157996e944db2906110d87ecb0c54" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:03:55.267275 24664 raft_consensus.cc:374] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:55.267547 24473 raft_consensus.cc:3037] T 37a157996e944db2906110d87ecb0c54 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:55.267588 24664 raft_consensus.cc:729] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Initialized, Role: FOLLOWER
I20260925 03:03:55.231567 24885 raft_consensus.cc:504] T 3d89396576884f7c8901dba54f783947 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:55.268364 24664 consensus_queue.cc:260] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:55.270495 24664 ts_tablet_manager.cc:1428] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5: Time spent starting tablet: real 0.018s user 0.008s sys 0.000s
I20260925 03:03:55.271288 24664 tablet_bootstrap.cc:492] T 550118bc1a0641ce9803bdbf43f2069f P 869b0a8575b54d08ae2817fad5dd42d5: Bootstrap starting.
I20260925 03:03:55.271116 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d89396576884f7c8901dba54f783947" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
W20260925 03:03:55.272439 24436 leader_election.cc:341] T 3d89396576884f7c8901dba54f783947 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:55.275425 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c23337a7cb74184a0e57b7095cb6a11" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:03:55.278707 24473 raft_consensus.cc:2454] T 37a157996e944db2906110d87ecb0c54 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 1.
I20260925 03:03:55.279326 24662 tablet_bootstrap.cc:654] T a4abe0ea757e4886be4945c5001e529a P eeab93abfdb141a895002ec9f312296c: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:55.284125 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d89396576884f7c8901dba54f783947" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:03:55.284821 24546 raft_consensus.cc:3037] T 3d89396576884f7c8901dba54f783947 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:55.288367 24663 tablet_bootstrap.cc:654] T 1414dcfe53f64a78ae413531eb8ec117 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:55.292429 24546 raft_consensus.cc:2454] T 3d89396576884f7c8901dba54f783947 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 1.
I20260925 03:03:55.293566 24885 leader_election.cc:288] T 3d89396576884f7c8901dba54f783947 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:55.295540 24664 tablet_bootstrap.cc:654] T 550118bc1a0641ce9803bdbf43f2069f P 869b0a8575b54d08ae2817fad5dd42d5: Neither blocks nor log segments found. Creating new log.
W20260925 03:03:55.296274 24514 leader_election.cc:341] T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:55.296778 24514 leader_election.cc:302] T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:03:55.298744 24820 raft_consensus.cc:2726] T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 03:03:55.302222 24514 leader_election.cc:302] T 37a157996e944db2906110d87ecb0c54 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:03:55.302929 24820 raft_consensus.cc:2781] T 37a157996e944db2906110d87ecb0c54 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:03:55.303474 24820 raft_consensus.cc:686] T 37a157996e944db2906110d87ecb0c54 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 LEADER]: Becoming Leader. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:03:55.304423 24820 consensus_queue.cc:237] T 37a157996e944db2906110d87ecb0c54 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:55.305454 24663 tablet_bootstrap.cc:492] T 1414dcfe53f64a78ae413531eb8ec117 P a04a37fc4aa7460cbfdbeeb4e451e1f5: No bootstrap required, opened a new log
I20260925 03:03:55.305895 24663 ts_tablet_manager.cc:1397] T 1414dcfe53f64a78ae413531eb8ec117 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent bootstrapping tablet: real 0.060s user 0.008s sys 0.002s
I20260925 03:03:55.308413 24663 raft_consensus.cc:348] T 1414dcfe53f64a78ae413531eb8ec117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:55.309096 24663 raft_consensus.cc:374] T 1414dcfe53f64a78ae413531eb8ec117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:55.309368 24663 raft_consensus.cc:729] T 1414dcfe53f64a78ae413531eb8ec117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Initialized, Role: FOLLOWER
I20260925 03:03:55.309525 24437 leader_election.cc:302] T 3d89396576884f7c8901dba54f783947 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:03:55.311347 24662 tablet_bootstrap.cc:492] T a4abe0ea757e4886be4945c5001e529a P eeab93abfdb141a895002ec9f312296c: No bootstrap required, opened a new log
I20260925 03:03:55.311833 24662 ts_tablet_manager.cc:1397] T a4abe0ea757e4886be4945c5001e529a P eeab93abfdb141a895002ec9f312296c: Time spent bootstrapping tablet: real 0.076s user 0.010s sys 0.017s
I20260925 03:03:55.312238 24885 raft_consensus.cc:2781] T 3d89396576884f7c8901dba54f783947 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:03:55.312655 24663 consensus_queue.cc:260] T 1414dcfe53f64a78ae413531eb8ec117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:55.314043 24814 raft_consensus.cc:482] T 1414dcfe53f64a78ae413531eb8ec117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:55.314488 24814 raft_consensus.cc:504] T 1414dcfe53f64a78ae413531eb8ec117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:55.315032 24662 raft_consensus.cc:348] T a4abe0ea757e4886be4945c5001e529a P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:55.316407 24814 leader_election.cc:288] T 1414dcfe53f64a78ae413531eb8ec117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:55.312737 24885 raft_consensus.cc:686] T 3d89396576884f7c8901dba54f783947 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 LEADER]: Becoming Leader. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:03:55.317730 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1414dcfe53f64a78ae413531eb8ec117" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
W20260925 03:03:55.319051 24511 leader_election.cc:341] T 1414dcfe53f64a78ae413531eb8ec117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:55.318639 24885 consensus_queue.cc:237] T 3d89396576884f7c8901dba54f783947 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:55.319892 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1414dcfe53f64a78ae413531eb8ec117" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
W20260925 03:03:55.321161 24514 leader_election.cc:341] T 1414dcfe53f64a78ae413531eb8ec117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:55.321552 24514 leader_election.cc:302] T 1414dcfe53f64a78ae413531eb8ec117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:03:55.322606 24662 raft_consensus.cc:374] T a4abe0ea757e4886be4945c5001e529a P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:55.322609 24763 raft_consensus.cc:2726] T 1414dcfe53f64a78ae413531eb8ec117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 03:03:55.323721 24662 raft_consensus.cc:729] T a4abe0ea757e4886be4945c5001e529a P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Initialized, Role: FOLLOWER
I20260925 03:03:55.325413 24662 consensus_queue.cc:260] T a4abe0ea757e4886be4945c5001e529a P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:55.319270 24663 ts_tablet_manager.cc:1428] T 1414dcfe53f64a78ae413531eb8ec117 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent starting tablet: real 0.013s user 0.003s sys 0.002s
I20260925 03:03:55.327257 24718 raft_consensus.cc:482] T b4ebd2b043024a548005c8b6b00b0449 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:55.328182 24718 raft_consensus.cc:504] T b4ebd2b043024a548005c8b6b00b0449 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:55.329344 24663 tablet_bootstrap.cc:492] T 0be4e0aa984f4f2b81a58e382e434ad5 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Bootstrap starting.
I20260925 03:03:55.330498 24718 leader_election.cc:288] T b4ebd2b043024a548005c8b6b00b0449 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:03:55.331319 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4ebd2b043024a548005c8b6b00b0449" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:03:55.332194 24471 raft_consensus.cc:2379] T b4ebd2b043024a548005c8b6b00b0449 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c in current term 1: Already voted for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in this term.
I20260925 03:03:55.333437 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4ebd2b043024a548005c8b6b00b0449" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:03:55.334234 24662 ts_tablet_manager.cc:1428] T a4abe0ea757e4886be4945c5001e529a P eeab93abfdb141a895002ec9f312296c: Time spent starting tablet: real 0.022s user 0.002s sys 0.004s
I20260925 03:03:55.335150 24662 tablet_bootstrap.cc:492] T a7aa47c48404465b94a36fd819fe3c7a P eeab93abfdb141a895002ec9f312296c: Bootstrap starting.
I20260925 03:03:55.335309 24586 leader_election.cc:302] T b4ebd2b043024a548005c8b6b00b0449 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:03:55.336647 24670 raft_consensus.cc:3037] T b4ebd2b043024a548005c8b6b00b0449 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:55.342628 24663 tablet_bootstrap.cc:654] T 0be4e0aa984f4f2b81a58e382e434ad5 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:55.351142 24670 raft_consensus.cc:2726] T b4ebd2b043024a548005c8b6b00b0449 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 03:03:55.352546 24662 tablet_bootstrap.cc:654] T a7aa47c48404465b94a36fd819fe3c7a P eeab93abfdb141a895002ec9f312296c: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:55.371088 24369 catalog_manager.cc:5495] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1). New cstate: current_term: 1 leader_uuid: "869b0a8575b54d08ae2817fad5dd42d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: UNKNOWN } } }
I20260925 03:03:55.380927 24371 catalog_manager.cc:5495] T 37a157996e944db2906110d87ecb0c54 P a04a37fc4aa7460cbfdbeeb4e451e1f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2). New cstate: current_term: 1 leader_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: UNKNOWN } } }
I20260925 03:03:55.410169 24663 tablet_bootstrap.cc:492] T 0be4e0aa984f4f2b81a58e382e434ad5 P a04a37fc4aa7460cbfdbeeb4e451e1f5: No bootstrap required, opened a new log
I20260925 03:03:55.410722 24663 ts_tablet_manager.cc:1397] T 0be4e0aa984f4f2b81a58e382e434ad5 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent bootstrapping tablet: real 0.083s user 0.011s sys 0.008s
I20260925 03:03:55.419220 24662 tablet_bootstrap.cc:492] T a7aa47c48404465b94a36fd819fe3c7a P eeab93abfdb141a895002ec9f312296c: No bootstrap required, opened a new log
I20260925 03:03:55.419711 24662 ts_tablet_manager.cc:1397] T a7aa47c48404465b94a36fd819fe3c7a P eeab93abfdb141a895002ec9f312296c: Time spent bootstrapping tablet: real 0.085s user 0.025s sys 0.036s
I20260925 03:03:55.419584 24663 raft_consensus.cc:348] T 0be4e0aa984f4f2b81a58e382e434ad5 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:55.420379 24663 raft_consensus.cc:374] T 0be4e0aa984f4f2b81a58e382e434ad5 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:55.420714 24663 raft_consensus.cc:729] T 0be4e0aa984f4f2b81a58e382e434ad5 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Initialized, Role: FOLLOWER
I20260925 03:03:55.421430 24663 consensus_queue.cc:260] T 0be4e0aa984f4f2b81a58e382e434ad5 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:55.420406 24664 tablet_bootstrap.cc:492] T 550118bc1a0641ce9803bdbf43f2069f P 869b0a8575b54d08ae2817fad5dd42d5: No bootstrap required, opened a new log
I20260925 03:03:55.422657 24664 ts_tablet_manager.cc:1397] T 550118bc1a0641ce9803bdbf43f2069f P 869b0a8575b54d08ae2817fad5dd42d5: Time spent bootstrapping tablet: real 0.152s user 0.034s sys 0.043s
I20260925 03:03:55.423509 24663 ts_tablet_manager.cc:1428] T 0be4e0aa984f4f2b81a58e382e434ad5 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent starting tablet: real 0.012s user 0.005s sys 0.002s
I20260925 03:03:55.424436 24663 tablet_bootstrap.cc:492] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5: Bootstrap starting.
I20260925 03:03:55.422099 24662 raft_consensus.cc:348] T a7aa47c48404465b94a36fd819fe3c7a P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:55.422060 24371 catalog_manager.cc:5495] T 3d89396576884f7c8901dba54f783947 P 869b0a8575b54d08ae2817fad5dd42d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1). New cstate: current_term: 1 leader_uuid: "869b0a8575b54d08ae2817fad5dd42d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: UNKNOWN } } }
I20260925 03:03:55.424943 24664 raft_consensus.cc:348] T 550118bc1a0641ce9803bdbf43f2069f P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:55.426496 24664 raft_consensus.cc:374] T 550118bc1a0641ce9803bdbf43f2069f P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:55.426733 24664 raft_consensus.cc:729] T 550118bc1a0641ce9803bdbf43f2069f P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Initialized, Role: FOLLOWER
I20260925 03:03:55.427304 24664 consensus_queue.cc:260] T 550118bc1a0641ce9803bdbf43f2069f P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:55.429729 24664 ts_tablet_manager.cc:1428] T 550118bc1a0641ce9803bdbf43f2069f P 869b0a8575b54d08ae2817fad5dd42d5: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
I20260925 03:03:55.430490 24664 tablet_bootstrap.cc:492] T 1414dcfe53f64a78ae413531eb8ec117 P 869b0a8575b54d08ae2817fad5dd42d5: Bootstrap starting.
I20260925 03:03:55.432880 24663 tablet_bootstrap.cc:654] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:55.425820 24662 raft_consensus.cc:374] T a7aa47c48404465b94a36fd819fe3c7a P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:55.435221 24662 raft_consensus.cc:729] T a7aa47c48404465b94a36fd819fe3c7a P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Initialized, Role: FOLLOWER
I20260925 03:03:55.444397 24664 tablet_bootstrap.cc:654] T 1414dcfe53f64a78ae413531eb8ec117 P 869b0a8575b54d08ae2817fad5dd42d5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:55.447230 24662 consensus_queue.cc:260] T a7aa47c48404465b94a36fd819fe3c7a P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:55.449169 24662 ts_tablet_manager.cc:1428] T a7aa47c48404465b94a36fd819fe3c7a P eeab93abfdb141a895002ec9f312296c: Time spent starting tablet: real 0.029s user 0.010s sys 0.005s
I20260925 03:03:55.450675 24662 tablet_bootstrap.cc:492] T 3d89396576884f7c8901dba54f783947 P eeab93abfdb141a895002ec9f312296c: Bootstrap starting.
I20260925 03:03:55.458396 24662 tablet_bootstrap.cc:654] T 3d89396576884f7c8901dba54f783947 P eeab93abfdb141a895002ec9f312296c: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:55.461938 24664 tablet_bootstrap.cc:492] T 1414dcfe53f64a78ae413531eb8ec117 P 869b0a8575b54d08ae2817fad5dd42d5: No bootstrap required, opened a new log
I20260925 03:03:55.462505 24664 ts_tablet_manager.cc:1397] T 1414dcfe53f64a78ae413531eb8ec117 P 869b0a8575b54d08ae2817fad5dd42d5: Time spent bootstrapping tablet: real 0.032s user 0.004s sys 0.008s
I20260925 03:03:55.465396 24664 raft_consensus.cc:348] T 1414dcfe53f64a78ae413531eb8ec117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:55.466904 24664 raft_consensus.cc:374] T 1414dcfe53f64a78ae413531eb8ec117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:55.467166 24664 raft_consensus.cc:729] T 1414dcfe53f64a78ae413531eb8ec117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Initialized, Role: FOLLOWER
I20260925 03:03:55.467718 24664 consensus_queue.cc:260] T 1414dcfe53f64a78ae413531eb8ec117 P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:55.469825 24662 tablet_bootstrap.cc:492] T 3d89396576884f7c8901dba54f783947 P eeab93abfdb141a895002ec9f312296c: No bootstrap required, opened a new log
I20260925 03:03:55.469921 24664 ts_tablet_manager.cc:1428] T 1414dcfe53f64a78ae413531eb8ec117 P 869b0a8575b54d08ae2817fad5dd42d5: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260925 03:03:55.470420 24662 ts_tablet_manager.cc:1397] T 3d89396576884f7c8901dba54f783947 P eeab93abfdb141a895002ec9f312296c: Time spent bootstrapping tablet: real 0.020s user 0.007s sys 0.011s
I20260925 03:03:55.472163 24664 tablet_bootstrap.cc:492] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5: Bootstrap starting.
I20260925 03:03:55.473104 24662 raft_consensus.cc:348] T 3d89396576884f7c8901dba54f783947 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:55.476387 24662 raft_consensus.cc:374] T 3d89396576884f7c8901dba54f783947 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:55.476724 24662 raft_consensus.cc:729] T 3d89396576884f7c8901dba54f783947 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Initialized, Role: FOLLOWER
I20260925 03:03:55.472311 24670 raft_consensus.cc:482] T a7aa47c48404465b94a36fd819fe3c7a P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:55.477415 24662 consensus_queue.cc:260] T 3d89396576884f7c8901dba54f783947 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:55.477959 24670 raft_consensus.cc:504] T a7aa47c48404465b94a36fd819fe3c7a P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:55.479776 24662 ts_tablet_manager.cc:1428] T 3d89396576884f7c8901dba54f783947 P eeab93abfdb141a895002ec9f312296c: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20260925 03:03:55.479991 24670 leader_election.cc:288] T a7aa47c48404465b94a36fd819fe3c7a P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:03:55.480677 24662 tablet_bootstrap.cc:492] T 367277daf8d04778b3fa4ed94d71580d P eeab93abfdb141a895002ec9f312296c: Bootstrap starting.
I20260925 03:03:55.480674 24664 tablet_bootstrap.cc:654] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:55.481863 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7aa47c48404465b94a36fd819fe3c7a" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:03:55.484933 24663 tablet_bootstrap.cc:492] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5: No bootstrap required, opened a new log
I20260925 03:03:55.485373 24663 ts_tablet_manager.cc:1397] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent bootstrapping tablet: real 0.061s user 0.009s sys 0.012s
I20260925 03:03:55.485935 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7aa47c48404465b94a36fd819fe3c7a" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:03:55.486501 24662 tablet_bootstrap.cc:654] T 367277daf8d04778b3fa4ed94d71580d P eeab93abfdb141a895002ec9f312296c: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:55.486732 24548 raft_consensus.cc:2379] T a7aa47c48404465b94a36fd819fe3c7a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c in current term 1: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:03:55.487830 24663 raft_consensus.cc:348] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:55.488956 24586 leader_election.cc:302] T a7aa47c48404465b94a36fd819fe3c7a P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:03:55.488517 24663 raft_consensus.cc:374] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:55.489921 24663 raft_consensus.cc:729] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Initialized, Role: FOLLOWER
I20260925 03:03:55.490095 24718 raft_consensus.cc:3037] T a7aa47c48404465b94a36fd819fe3c7a P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:55.490556 24663 consensus_queue.cc:260] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:55.492636 24663 ts_tablet_manager.cc:1428] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20260925 03:03:55.496296 24622 raft_consensus.cc:3037] T a4abe0ea757e4886be4945c5001e529a P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:55.513087 24763 consensus_queue.cc:1035] T a4abe0ea757e4886be4945c5001e529a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260925 03:03:55.514035 24663 tablet_bootstrap.cc:492] T 550118bc1a0641ce9803bdbf43f2069f P a04a37fc4aa7460cbfdbeeb4e451e1f5: Bootstrap starting.
I20260925 03:03:55.521934 24663 tablet_bootstrap.cc:654] T 550118bc1a0641ce9803bdbf43f2069f P a04a37fc4aa7460cbfdbeeb4e451e1f5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:55.524405 24718 raft_consensus.cc:2726] T a7aa47c48404465b94a36fd819fe3c7a P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 03:03:55.538434 24879 consensus_queue.cc:1035] T a7aa47c48404465b94a36fd819fe3c7a P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 03:03:55.538864 24663 tablet_bootstrap.cc:492] T 550118bc1a0641ce9803bdbf43f2069f P a04a37fc4aa7460cbfdbeeb4e451e1f5: No bootstrap required, opened a new log
I20260925 03:03:55.539384 24663 ts_tablet_manager.cc:1397] T 550118bc1a0641ce9803bdbf43f2069f P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent bootstrapping tablet: real 0.026s user 0.016s sys 0.000s
I20260925 03:03:55.538864 24664 tablet_bootstrap.cc:492] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5: No bootstrap required, opened a new log
I20260925 03:03:55.541762 24664 ts_tablet_manager.cc:1397] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5: Time spent bootstrapping tablet: real 0.070s user 0.029s sys 0.013s
I20260925 03:03:55.542402 24663 raft_consensus.cc:348] T 550118bc1a0641ce9803bdbf43f2069f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:55.543089 24663 raft_consensus.cc:374] T 550118bc1a0641ce9803bdbf43f2069f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:55.543352 24663 raft_consensus.cc:729] T 550118bc1a0641ce9803bdbf43f2069f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Initialized, Role: FOLLOWER
I20260925 03:03:55.545372 24662 tablet_bootstrap.cc:492] T 367277daf8d04778b3fa4ed94d71580d P eeab93abfdb141a895002ec9f312296c: No bootstrap required, opened a new log
I20260925 03:03:55.545877 24662 ts_tablet_manager.cc:1397] T 367277daf8d04778b3fa4ed94d71580d P eeab93abfdb141a895002ec9f312296c: Time spent bootstrapping tablet: real 0.065s user 0.019s sys 0.034s
I20260925 03:03:55.544029 24663 consensus_queue.cc:260] T 550118bc1a0641ce9803bdbf43f2069f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:55.554778 24664 raft_consensus.cc:348] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:55.555670 24664 raft_consensus.cc:374] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:55.555999 24664 raft_consensus.cc:729] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Initialized, Role: FOLLOWER
I20260925 03:03:55.553032 24879 consensus_queue.cc:1035] T a7aa47c48404465b94a36fd819fe3c7a P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260925 03:03:55.559806 24814 consensus_queue.cc:1035] T a4abe0ea757e4886be4945c5001e529a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260925 03:03:55.563196 24663 ts_tablet_manager.cc:1428] T 550118bc1a0641ce9803bdbf43f2069f P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent starting tablet: real 0.024s user 0.006s sys 0.000s
I20260925 03:03:55.564203 24663 tablet_bootstrap.cc:492] T 367277daf8d04778b3fa4ed94d71580d P a04a37fc4aa7460cbfdbeeb4e451e1f5: Bootstrap starting.
I20260925 03:03:55.565610 24664 consensus_queue.cc:260] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:55.566036 24662 raft_consensus.cc:348] T 367277daf8d04778b3fa4ed94d71580d P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:55.566758 24662 raft_consensus.cc:374] T 367277daf8d04778b3fa4ed94d71580d P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:55.567028 24662 raft_consensus.cc:729] T 367277daf8d04778b3fa4ed94d71580d P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Initialized, Role: FOLLOWER
I20260925 03:03:55.568189 24664 ts_tablet_manager.cc:1428] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5: Time spent starting tablet: real 0.026s user 0.004s sys 0.003s
I20260925 03:03:55.567828 24662 consensus_queue.cc:260] T 367277daf8d04778b3fa4ed94d71580d P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:55.569829 24820 raft_consensus.cc:482] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:55.578300 24820 raft_consensus.cc:504] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:55.579609 24662 ts_tablet_manager.cc:1428] T 367277daf8d04778b3fa4ed94d71580d P eeab93abfdb141a895002ec9f312296c: Time spent starting tablet: real 0.033s user 0.007s sys 0.000s
I20260925 03:03:55.579878 24663 tablet_bootstrap.cc:654] T 367277daf8d04778b3fa4ed94d71580d P a04a37fc4aa7460cbfdbeeb4e451e1f5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:55.575779 24763 consensus_queue.cc:1035] T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 03:03:55.580605 24820 leader_election.cc:288] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:55.580538 24662 tablet_bootstrap.cc:492] T 1414dcfe53f64a78ae413531eb8ec117 P eeab93abfdb141a895002ec9f312296c: Bootstrap starting.
I20260925 03:03:55.581683 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15861cd28e354598a3c6bfc4337c34bb" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:03:55.576713 24664 tablet_bootstrap.cc:492] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5: Bootstrap starting.
I20260925 03:03:55.581789 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15861cd28e354598a3c6bfc4337c34bb" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
W20260925 03:03:55.584465 24514 leader_election.cc:341] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187): Illegal state: must be running to vote when last-logged opid is not known
W20260925 03:03:55.585753 24511 leader_election.cc:341] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:55.586146 24511 leader_election.cc:302] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:03:55.587167 24820 raft_consensus.cc:2726] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 03:03:55.607736 24820 consensus_queue.cc:1035] T b4ebd2b043024a548005c8b6b00b0449 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260925 03:03:55.609193 24664 tablet_bootstrap.cc:654] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:55.634404 24663 tablet_bootstrap.cc:492] T 367277daf8d04778b3fa4ed94d71580d P a04a37fc4aa7460cbfdbeeb4e451e1f5: No bootstrap required, opened a new log
I20260925 03:03:55.634927 24663 ts_tablet_manager.cc:1397] T 367277daf8d04778b3fa4ed94d71580d P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent bootstrapping tablet: real 0.071s user 0.011s sys 0.012s
I20260925 03:03:55.637281 24763 consensus_queue.cc:1035] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 03:03:55.645007 24708 consensus_queue.cc:1035] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260925 03:03:55.702541 24663 raft_consensus.cc:348] T 367277daf8d04778b3fa4ed94d71580d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:55.703330 24663 raft_consensus.cc:374] T 367277daf8d04778b3fa4ed94d71580d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:55.703589 24663 raft_consensus.cc:729] T 367277daf8d04778b3fa4ed94d71580d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Initialized, Role: FOLLOWER
I20260925 03:03:55.704185 24663 consensus_queue.cc:260] T 367277daf8d04778b3fa4ed94d71580d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:55.707144 24870 raft_consensus.cc:482] T 0be4e0aa984f4f2b81a58e382e434ad5 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:55.709856 24879 consensus_queue.cc:1035] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 03:03:55.712360 24662 tablet_bootstrap.cc:654] T 1414dcfe53f64a78ae413531eb8ec117 P eeab93abfdb141a895002ec9f312296c: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:55.709270 24870 raft_consensus.cc:504] T 0be4e0aa984f4f2b81a58e382e434ad5 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:55.715346 24880 raft_consensus.cc:482] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:55.730865 24880 raft_consensus.cc:504] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:55.736120 24880 leader_election.cc:288] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:55.750576 24870 leader_election.cc:288] T 0be4e0aa984f4f2b81a58e382e434ad5 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:55.755934 24663 ts_tablet_manager.cc:1428] T 367277daf8d04778b3fa4ed94d71580d P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent starting tablet: real 0.121s user 0.014s sys 0.007s
I20260925 03:03:55.762569 24663 tablet_bootstrap.cc:492] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5: Bootstrap starting.
I20260925 03:03:55.765066 24664 tablet_bootstrap.cc:492] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5: No bootstrap required, opened a new log
I20260925 03:03:55.765656 24664 ts_tablet_manager.cc:1397] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5: Time spent bootstrapping tablet: real 0.189s user 0.007s sys 0.041s
I20260925 03:03:55.773329 24663 tablet_bootstrap.cc:654] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:55.785053 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367277daf8d04778b3fa4ed94d71580d" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:03:55.785720 24622 raft_consensus.cc:2454] T 367277daf8d04778b3fa4ed94d71580d P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 0.
I20260925 03:03:55.789285 24436 leader_election.cc:302] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:03:55.788719 24664 raft_consensus.cc:348] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:55.790295 24708 raft_consensus.cc:2781] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 03:03:55.790678 24708 raft_consensus.cc:482] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:03:55.791004 24708 raft_consensus.cc:3037] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:55.793929 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be4e0aa984f4f2b81a58e382e434ad5" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:03:55.790012 24664 raft_consensus.cc:374] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:55.796960 24664 raft_consensus.cc:729] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Initialized, Role: FOLLOWER
I20260925 03:03:55.797677 24664 consensus_queue.cc:260] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:55.799295 24708 raft_consensus.cc:504] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:55.812572 24664 ts_tablet_manager.cc:1428] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5: Time spent starting tablet: real 0.047s user 0.007s sys 0.000s
I20260925 03:03:55.812718 24662 tablet_bootstrap.cc:492] T 1414dcfe53f64a78ae413531eb8ec117 P eeab93abfdb141a895002ec9f312296c: No bootstrap required, opened a new log
I20260925 03:03:55.812862 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be4e0aa984f4f2b81a58e382e434ad5" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:03:55.813289 24662 ts_tablet_manager.cc:1397] T 1414dcfe53f64a78ae413531eb8ec117 P eeab93abfdb141a895002ec9f312296c: Time spent bootstrapping tablet: real 0.233s user 0.013s sys 0.015s
I20260925 03:03:55.813552 24664 tablet_bootstrap.cc:492] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5: Bootstrap starting.
I20260925 03:03:55.815550 24929 raft_consensus.cc:482] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:55.819379 24664 tablet_bootstrap.cc:654] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:55.819880 24931 consensus_queue.cc:1035] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260925 03:03:55.822103 24708 leader_election.cc:288] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:55.823144 24923 consensus_queue.cc:1035] T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260925 03:03:55.823912 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367277daf8d04778b3fa4ed94d71580d" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:03:55.824595 24619 raft_consensus.cc:3037] T 367277daf8d04778b3fa4ed94d71580d P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Advancing to term 1
W20260925 03:03:55.824101 24514 leader_election.cc:341] T 0be4e0aa984f4f2b81a58e382e434ad5 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:55.827025 24663 tablet_bootstrap.cc:492] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5: No bootstrap required, opened a new log
I20260925 03:03:55.827512 24663 ts_tablet_manager.cc:1397] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent bootstrapping tablet: real 0.065s user 0.009s sys 0.003s
I20260925 03:03:55.821782 24929 raft_consensus.cc:504] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:55.823951 24662 raft_consensus.cc:348] T 1414dcfe53f64a78ae413531eb8ec117 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:55.833021 24662 raft_consensus.cc:374] T 1414dcfe53f64a78ae413531eb8ec117 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:55.833443 24662 raft_consensus.cc:729] T 1414dcfe53f64a78ae413531eb8ec117 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Initialized, Role: FOLLOWER
I20260925 03:03:55.834244 24662 consensus_queue.cc:260] T 1414dcfe53f64a78ae413531eb8ec117 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:55.835130 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c23337a7cb74184a0e57b7095cb6a11" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:03:55.836077 24929 leader_election.cc:288] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
W20260925 03:03:55.840411 24436 leader_election.cc:341] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:55.842964 24619 raft_consensus.cc:2454] T 367277daf8d04778b3fa4ed94d71580d P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 1.
I20260925 03:03:55.843951 24662 ts_tablet_manager.cc:1428] T 1414dcfe53f64a78ae413531eb8ec117 P eeab93abfdb141a895002ec9f312296c: Time spent starting tablet: real 0.030s user 0.006s sys 0.000s
I20260925 03:03:55.844139 24436 leader_election.cc:302] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:03:55.846323 24929 raft_consensus.cc:2781] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:03:55.844897 24663 raft_consensus.cc:348] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:55.846982 24929 raft_consensus.cc:686] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 LEADER]: Becoming Leader. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:03:55.846520 24662 tablet_bootstrap.cc:492] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c: Bootstrap starting.
I20260925 03:03:55.848100 24929 consensus_queue.cc:237] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:55.847478 24663 raft_consensus.cc:374] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:55.849529 24663 raft_consensus.cc:729] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Initialized, Role: FOLLOWER
I20260925 03:03:55.850423 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367277daf8d04778b3fa4ed94d71580d" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:03:55.853857 24548 raft_consensus.cc:2454] T 367277daf8d04778b3fa4ed94d71580d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 0.
I20260925 03:03:55.854620 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367277daf8d04778b3fa4ed94d71580d" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:03:55.855212 24547 raft_consensus.cc:3037] T 367277daf8d04778b3fa4ed94d71580d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:55.852084 24923 consensus_queue.cc:1035] T b4ebd2b043024a548005c8b6b00b0449 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 03:03:55.854940 24663 consensus_queue.cc:260] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:55.857647 24619 raft_consensus.cc:3037] T 3d89396576884f7c8901dba54f783947 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:55.863018 24663 ts_tablet_manager.cc:1428] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent starting tablet: real 0.035s user 0.004s sys 0.004s
I20260925 03:03:55.863312 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c23337a7cb74184a0e57b7095cb6a11" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:03:55.868849 24548 raft_consensus.cc:2454] T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 0.
I20260925 03:03:55.870177 24663 tablet_bootstrap.cc:492] T ac9357d144aa49789faaf31fe5e6bf89 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Bootstrap starting.
I20260925 03:03:55.871888 24929 consensus_queue.cc:1035] T 3d89396576884f7c8901dba54f783947 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260925 03:03:55.867143 24547 raft_consensus.cc:2454] T 367277daf8d04778b3fa4ed94d71580d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 1.
I20260925 03:03:55.870591 24664 tablet_bootstrap.cc:492] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5: No bootstrap required, opened a new log
I20260925 03:03:55.874476 24664 ts_tablet_manager.cc:1397] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5: Time spent bootstrapping tablet: real 0.061s user 0.009s sys 0.005s
I20260925 03:03:55.877439 24664 raft_consensus.cc:348] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:55.878153 24664 raft_consensus.cc:374] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:55.878528 24664 raft_consensus.cc:729] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Initialized, Role: FOLLOWER
I20260925 03:03:55.878147 24920 consensus_queue.cc:1035] T 37a157996e944db2906110d87ecb0c54 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260925 03:03:55.861285 24662 tablet_bootstrap.cc:654] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c: Neither blocks nor log segments found. Creating new log.
W20260925 03:03:55.879251 24511 leader_election.cc:341] T 0be4e0aa984f4f2b81a58e382e434ad5 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:55.879683 24511 leader_election.cc:302] T 0be4e0aa984f4f2b81a58e382e434ad5 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:03:55.880481 24663 tablet_bootstrap.cc:654] T ac9357d144aa49789faaf31fe5e6bf89 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:55.882202 24664 consensus_queue.cc:260] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:55.887969 24664 ts_tablet_manager.cc:1428] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5: Time spent starting tablet: real 0.013s user 0.004s sys 0.004s
I20260925 03:03:55.888984 24664 tablet_bootstrap.cc:492] T 21c43af5e6844fdaacc8c625fa01fd8d P 869b0a8575b54d08ae2817fad5dd42d5: Bootstrap starting.
I20260925 03:03:55.892985 24921 raft_consensus.cc:2726] T 0be4e0aa984f4f2b81a58e382e434ad5 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 03:03:55.894865 24663 tablet_bootstrap.cc:492] T ac9357d144aa49789faaf31fe5e6bf89 P a04a37fc4aa7460cbfdbeeb4e451e1f5: No bootstrap required, opened a new log
I20260925 03:03:55.899914 24662 tablet_bootstrap.cc:492] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c: No bootstrap required, opened a new log
I20260925 03:03:55.900413 24662 ts_tablet_manager.cc:1397] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c: Time spent bootstrapping tablet: real 0.054s user 0.008s sys 0.002s
I20260925 03:03:55.895319 24663 ts_tablet_manager.cc:1397] T ac9357d144aa49789faaf31fe5e6bf89 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.014s
I20260925 03:03:55.902958 24662 raft_consensus.cc:348] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:55.903638 24662 raft_consensus.cc:374] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:55.903466 24663 raft_consensus.cc:348] T ac9357d144aa49789faaf31fe5e6bf89 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:55.904029 24662 raft_consensus.cc:729] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Initialized, Role: FOLLOWER
I20260925 03:03:55.904258 24663 raft_consensus.cc:374] T ac9357d144aa49789faaf31fe5e6bf89 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:55.905517 24663 raft_consensus.cc:729] T ac9357d144aa49789faaf31fe5e6bf89 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Initialized, Role: FOLLOWER
I20260925 03:03:55.904675 24662 consensus_queue.cc:260] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:03:55.883195 24418 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260925 03:03:55.906464 24663 consensus_queue.cc:260] T ac9357d144aa49789faaf31fe5e6bf89 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:55.941895 24928 raft_consensus.cc:482] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:55.942607 24663 ts_tablet_manager.cc:1428] T ac9357d144aa49789faaf31fe5e6bf89 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent starting tablet: real 0.042s user 0.010s sys 0.015s
I20260925 03:03:55.942513 24928 raft_consensus.cc:504] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:55.943917 24708 consensus_queue.cc:1035] T 3d89396576884f7c8901dba54f783947 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260925 03:03:55.946880 24663 tablet_bootstrap.cc:492] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Bootstrap starting.
I20260925 03:03:55.946897 24437 leader_election.cc:302] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:03:55.948666 24879 raft_consensus.cc:482] T 550118bc1a0641ce9803bdbf43f2069f P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:55.949209 24879 raft_consensus.cc:504] T 550118bc1a0641ce9803bdbf43f2069f P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:55.955837 24662 ts_tablet_manager.cc:1428] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c: Time spent starting tablet: real 0.055s user 0.008s sys 0.000s
I20260925 03:03:55.957134 24662 tablet_bootstrap.cc:492] T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c: Bootstrap starting.
I20260925 03:03:55.956085 24885 raft_consensus.cc:2781] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 03:03:55.949231 24708 raft_consensus.cc:482] T 1414dcfe53f64a78ae413531eb8ec117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:55.956439 24921 raft_consensus.cc:482] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:55.961090 24371 catalog_manager.cc:5495] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1). New cstate: current_term: 1 leader_uuid: "869b0a8575b54d08ae2817fad5dd42d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: UNKNOWN } } }
I20260925 03:03:55.956879 24879 leader_election.cc:288] T 550118bc1a0641ce9803bdbf43f2069f P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:55.972573 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "550118bc1a0641ce9803bdbf43f2069f" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:03:55.965471 24885 raft_consensus.cc:482] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:03:55.968010 24708 raft_consensus.cc:504] T 1414dcfe53f64a78ae413531eb8ec117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:55.974951 24885 raft_consensus.cc:3037] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:55.982069 24920 raft_consensus.cc:482] T 550118bc1a0641ce9803bdbf43f2069f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:55.975844 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "550118bc1a0641ce9803bdbf43f2069f" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:03:55.996522 24920 raft_consensus.cc:504] T 550118bc1a0641ce9803bdbf43f2069f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:55.998524 24708 leader_election.cc:288] T 1414dcfe53f64a78ae413531eb8ec117 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:55.998724 24548 raft_consensus.cc:2454] T 550118bc1a0641ce9803bdbf43f2069f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 0.
I20260925 03:03:55.998584 24920 leader_election.cc:288] T 550118bc1a0641ce9803bdbf43f2069f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:55.970186 24921 raft_consensus.cc:504] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:03:55.980970 24436 leader_election.cc:341] T 550118bc1a0641ce9803bdbf43f2069f P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:56.004245 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1414dcfe53f64a78ae413531eb8ec117" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:03:56.004920 24548 raft_consensus.cc:2454] T 1414dcfe53f64a78ae413531eb8ec117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 0.
I20260925 03:03:56.009454 24928 leader_election.cc:288] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:03:56.016379 24437 leader_election.cc:302] T 550118bc1a0641ce9803bdbf43f2069f P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:03:56.018323 24879 raft_consensus.cc:2781] T 550118bc1a0641ce9803bdbf43f2069f P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 03:03:56.018708 24879 raft_consensus.cc:482] T 550118bc1a0641ce9803bdbf43f2069f P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:03:56.019014 24879 raft_consensus.cc:3037] T 550118bc1a0641ce9803bdbf43f2069f P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:56.020736 24437 leader_election.cc:302] T 1414dcfe53f64a78ae413531eb8ec117 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:03:56.012869 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c23337a7cb74184a0e57b7095cb6a11" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:03:56.021644 24708 raft_consensus.cc:2781] T 1414dcfe53f64a78ae413531eb8ec117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 03:03:56.022006 24708 raft_consensus.cc:482] T 1414dcfe53f64a78ae413531eb8ec117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:03:56.022330 24708 raft_consensus.cc:3037] T 1414dcfe53f64a78ae413531eb8ec117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:56.022377 24921 leader_election.cc:288] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:56.021641 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c23337a7cb74184a0e57b7095cb6a11" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:03:56.024796 24546 raft_consensus.cc:2454] T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 0.
I20260925 03:03:56.025899 24586 leader_election.cc:302] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:03:56.026726 24717 raft_consensus.cc:2781] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 03:03:56.027058 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1414dcfe53f64a78ae413531eb8ec117" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:03:56.026988 24885 raft_consensus.cc:504] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:56.027693 24622 raft_consensus.cc:2454] T 1414dcfe53f64a78ae413531eb8ec117 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 0.
I20260925 03:03:56.027079 24717 raft_consensus.cc:482] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:03:56.029086 24717 raft_consensus.cc:3037] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:56.033033 24471 raft_consensus.cc:2379] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c in current term 1: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:03:56.033279 24879 raft_consensus.cc:504] T 550118bc1a0641ce9803bdbf43f2069f P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:56.032660 24708 raft_consensus.cc:504] T 1414dcfe53f64a78ae413531eb8ec117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:56.035203 24879 leader_election.cc:288] T 550118bc1a0641ce9803bdbf43f2069f P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:56.038694 24708 leader_election.cc:288] T 1414dcfe53f64a78ae413531eb8ec117 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:56.039510 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "550118bc1a0641ce9803bdbf43f2069f" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:03:56.039710 24708 raft_consensus.cc:482] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:56.041055 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1414dcfe53f64a78ae413531eb8ec117" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:03:56.041630 24617 raft_consensus.cc:3037] T 1414dcfe53f64a78ae413531eb8ec117 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:56.041851 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "550118bc1a0641ce9803bdbf43f2069f" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:03:56.046315 24664 tablet_bootstrap.cc:654] T 21c43af5e6844fdaacc8c625fa01fd8d P 869b0a8575b54d08ae2817fad5dd42d5: Neither blocks nor log segments found. Creating new log.
W20260925 03:03:56.048270 24511 consensus_peers.cc:487] T 37a157996e944db2906110d87ecb0c54 P a04a37fc4aa7460cbfdbeeb4e451e1f5 -> Peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105): Couldn't send request to peer eeab93abfdb141a895002ec9f312296c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20260925 03:03:56.049067 24511 leader_election.cc:341] T 550118bc1a0641ce9803bdbf43f2069f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:56.051877 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c23337a7cb74184a0e57b7095cb6a11" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:03:56.052567 24548 raft_consensus.cc:3037] T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:56.055065 24617 raft_consensus.cc:2454] T 1414dcfe53f64a78ae413531eb8ec117 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 1.
I20260925 03:03:56.051684 24717 raft_consensus.cc:504] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:56.040570 24708 raft_consensus.cc:504] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:03:56.049155 24436 leader_election.cc:341] T 550118bc1a0641ce9803bdbf43f2069f P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:56.050236 24885 leader_election.cc:288] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:56.062395 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1414dcfe53f64a78ae413531eb8ec117" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:03:56.063061 24544 raft_consensus.cc:3037] T 1414dcfe53f64a78ae413531eb8ec117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:56.059126 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "550118bc1a0641ce9803bdbf43f2069f" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:03:56.064261 24473 raft_consensus.cc:2379] T 550118bc1a0641ce9803bdbf43f2069f P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 1: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:03:56.065789 24708 leader_election.cc:288] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:56.066334 24717 leader_election.cc:288] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 1 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:03:56.062394 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "550118bc1a0641ce9803bdbf43f2069f" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:03:56.069346 24546 raft_consensus.cc:3037] T 550118bc1a0641ce9803bdbf43f2069f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:56.069236 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c23337a7cb74184a0e57b7095cb6a11" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:03:56.070858 24663 tablet_bootstrap.cc:654] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:56.068264 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15861cd28e354598a3c6bfc4337c34bb" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:03:56.080873 24543 raft_consensus.cc:2454] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 0.
I20260925 03:03:56.083063 24514 leader_election.cc:302] T 550118bc1a0641ce9803bdbf43f2069f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:03:56.086544 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f1fac028a144058b92331c0d9b44ebf" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:03:56.089869 24546 raft_consensus.cc:2454] T 550118bc1a0641ce9803bdbf43f2069f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 1.
I20260925 03:03:56.090359 24870 raft_consensus.cc:2726] T 550118bc1a0641ce9803bdbf43f2069f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 03:03:56.079879 24662 tablet_bootstrap.cc:654] T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:56.093926 24437 leader_election.cc:302] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:03:56.107331 24437 leader_election.cc:302] T 550118bc1a0641ce9803bdbf43f2069f P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:03:56.108253 24885 raft_consensus.cc:2781] T 550118bc1a0641ce9803bdbf43f2069f P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:03:56.098232 24879 raft_consensus.cc:2781] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 03:03:56.080005 24544 raft_consensus.cc:2454] T 1414dcfe53f64a78ae413531eb8ec117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 1.
I20260925 03:03:56.115538 24814 raft_consensus.cc:482] T ac9357d144aa49789faaf31fe5e6bf89 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:56.108646 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c23337a7cb74184a0e57b7095cb6a11" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:03:56.116098 24814 raft_consensus.cc:504] T ac9357d144aa49789faaf31fe5e6bf89 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:56.117005 24617 raft_consensus.cc:2379] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 in current term 1: Already voted for candidate eeab93abfdb141a895002ec9f312296c in this term.
I20260925 03:03:56.118100 24814 leader_election.cc:288] T ac9357d144aa49789faaf31fe5e6bf89 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:56.109733 24879 raft_consensus.cc:482] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:03:56.120141 24879 raft_consensus.cc:3037] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:56.111316 24885 raft_consensus.cc:686] T 550118bc1a0641ce9803bdbf43f2069f P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 LEADER]: Becoming Leader. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:03:56.124750 24885 consensus_queue.cc:237] T 550118bc1a0641ce9803bdbf43f2069f P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:56.126650 24879 raft_consensus.cc:504] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:03:56.109313 24514 leader_election.cc:341] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:56.111756 24436 leader_election.cc:302] T 1414dcfe53f64a78ae413531eb8ec117 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:03:56.130697 24880 raft_consensus.cc:2781] T 1414dcfe53f64a78ae413531eb8ec117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:03:56.130983 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f1fac028a144058b92331c0d9b44ebf" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:03:56.131182 24880 raft_consensus.cc:686] T 1414dcfe53f64a78ae413531eb8ec117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 LEADER]: Becoming Leader. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:03:56.131880 24879 leader_election.cc:288] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:56.132076 24880 consensus_queue.cc:237] T 1414dcfe53f64a78ae413531eb8ec117 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:56.132579 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15861cd28e354598a3c6bfc4337c34bb" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:03:56.133193 24544 raft_consensus.cc:3037] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:56.139407 24664 tablet_bootstrap.cc:492] T 21c43af5e6844fdaacc8c625fa01fd8d P 869b0a8575b54d08ae2817fad5dd42d5: No bootstrap required, opened a new log
I20260925 03:03:56.139818 24664 ts_tablet_manager.cc:1397] T 21c43af5e6844fdaacc8c625fa01fd8d P 869b0a8575b54d08ae2817fad5dd42d5: Time spent bootstrapping tablet: real 0.251s user 0.009s sys 0.020s
I20260925 03:03:56.143639 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15861cd28e354598a3c6bfc4337c34bb" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:03:56.149359 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15861cd28e354598a3c6bfc4337c34bb" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
W20260925 03:03:56.151129 24436 leader_election.cc:341] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105): Illegal state: must be running to vote when last-logged opid is not known
W20260925 03:03:56.152040 24436 leader_election.cc:341] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105): Illegal state: must be running to vote when last-logged opid is not known
W20260925 03:03:56.155458 24511 leader_election.cc:341] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:56.155874 24511 leader_election.cc:302] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:03:56.156941 24920 raft_consensus.cc:2726] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 03:03:56.141233 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9357d144aa49789faaf31fe5e6bf89" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:03:56.153878 24544 raft_consensus.cc:2454] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 1.
I20260925 03:03:56.167038 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9357d144aa49789faaf31fe5e6bf89" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
W20260925 03:03:56.176795 24514 leader_election.cc:341] T ac9357d144aa49789faaf31fe5e6bf89 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:56.214758 24662 tablet_bootstrap.cc:492] T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c: No bootstrap required, opened a new log
I20260925 03:03:56.214758 24663 tablet_bootstrap.cc:492] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5: No bootstrap required, opened a new log
I20260925 03:03:56.215713 24662 ts_tablet_manager.cc:1397] T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c: Time spent bootstrapping tablet: real 0.259s user 0.033s sys 0.017s
I20260925 03:03:56.215710 24663 ts_tablet_manager.cc:1397] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent bootstrapping tablet: real 0.269s user 0.020s sys 0.032s
I20260925 03:03:56.215862 24437 leader_election.cc:302] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:03:56.217694 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c23337a7cb74184a0e57b7095cb6a11" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:03:56.218873 24708 raft_consensus.cc:2781] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:03:56.220055 24473 raft_consensus.cc:2379] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c in current term 1: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:03:56.220239 24708 raft_consensus.cc:686] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 LEADER]: Becoming Leader. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:03:56.221068 24708 consensus_queue.cc:237] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:56.224031 24548 raft_consensus.cc:2454] T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 1.
I20260925 03:03:56.224849 24664 raft_consensus.cc:348] T 21c43af5e6844fdaacc8c625fa01fd8d P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:56.225567 24664 raft_consensus.cc:374] T 21c43af5e6844fdaacc8c625fa01fd8d P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:56.225837 24664 raft_consensus.cc:729] T 21c43af5e6844fdaacc8c625fa01fd8d P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Initialized, Role: FOLLOWER
I20260925 03:03:56.226424 24664 consensus_queue.cc:260] T 21c43af5e6844fdaacc8c625fa01fd8d P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:56.228124 24437 leader_election.cc:302] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:03:56.229122 24879 raft_consensus.cc:2781] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:03:56.229012 24588 leader_election.cc:302] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:03:56.229660 24879 raft_consensus.cc:686] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 LEADER]: Becoming Leader. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:03:56.218426 24880 raft_consensus.cc:482] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:56.231563 24879 consensus_queue.cc:237] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:56.232944 24717 raft_consensus.cc:2726] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260925 03:03:56.232715 24663 raft_consensus.cc:348] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:56.233958 24663 raft_consensus.cc:374] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:56.234233 24663 raft_consensus.cc:729] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Initialized, Role: FOLLOWER
W20260925 03:03:56.235000 24511 leader_election.cc:341] T ac9357d144aa49789faaf31fe5e6bf89 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:56.235384 24511 leader_election.cc:302] T ac9357d144aa49789faaf31fe5e6bf89 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:03:56.236541 24814 raft_consensus.cc:2726] T ac9357d144aa49789faaf31fe5e6bf89 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 03:03:56.228534 24664 ts_tablet_manager.cc:1428] T 21c43af5e6844fdaacc8c625fa01fd8d P 869b0a8575b54d08ae2817fad5dd42d5: Time spent starting tablet: real 0.089s user 0.003s sys 0.000s
I20260925 03:03:56.234915 24663 consensus_queue.cc:260] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:56.238749 24664 tablet_bootstrap.cc:492] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5: Bootstrap starting.
I20260925 03:03:56.232185 24880 raft_consensus.cc:504] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:56.241951 24663 ts_tablet_manager.cc:1428] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent starting tablet: real 0.026s user 0.005s sys 0.001s
I20260925 03:03:56.245848 24663 tablet_bootstrap.cc:492] T 11bb46bce3564841b7674e6a693bd2e6 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Bootstrap starting.
I20260925 03:03:56.251536 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:03:56.252104 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:03:56.254618 24662 raft_consensus.cc:348] T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:56.255143 24662 raft_consensus.cc:374] T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:56.255362 24662 raft_consensus.cc:729] T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Initialized, Role: FOLLOWER
I20260925 03:03:56.255911 24662 consensus_queue.cc:260] T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:56.252765 24544 raft_consensus.cc:2454] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 0.
W20260925 03:03:56.257247 24436 leader_election.cc:341] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:56.252751 24880 leader_election.cc:288] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:56.258289 24437 leader_election.cc:302] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:03:56.261351 24880 raft_consensus.cc:2781] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 03:03:56.261813 24663 tablet_bootstrap.cc:654] T 11bb46bce3564841b7674e6a693bd2e6 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:56.266256 24662 ts_tablet_manager.cc:1428] T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c: Time spent starting tablet: real 0.050s user 0.003s sys 0.002s
I20260925 03:03:56.261808 24880 raft_consensus.cc:482] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:03:56.267149 24662 tablet_bootstrap.cc:492] T 37a157996e944db2906110d87ecb0c54 P eeab93abfdb141a895002ec9f312296c: Bootstrap starting.
I20260925 03:03:56.272974 24880 raft_consensus.cc:3037] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:56.288189 24368 catalog_manager.cc:5495] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1). New cstate: current_term: 1 leader_uuid: "869b0a8575b54d08ae2817fad5dd42d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: UNKNOWN } } }
I20260925 03:03:56.291618 24662 tablet_bootstrap.cc:654] T 37a157996e944db2906110d87ecb0c54 P eeab93abfdb141a895002ec9f312296c: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:56.293169 24368 catalog_manager.cc:5495] T 1414dcfe53f64a78ae413531eb8ec117 P 869b0a8575b54d08ae2817fad5dd42d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1). New cstate: current_term: 1 leader_uuid: "869b0a8575b54d08ae2817fad5dd42d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: UNKNOWN } } }
I20260925 03:03:56.294430 24368 catalog_manager.cc:5495] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1). New cstate: current_term: 1 leader_uuid: "869b0a8575b54d08ae2817fad5dd42d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: UNKNOWN } } }
I20260925 03:03:56.295961 24368 catalog_manager.cc:5495] T 550118bc1a0641ce9803bdbf43f2069f P 869b0a8575b54d08ae2817fad5dd42d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1). New cstate: current_term: 1 leader_uuid: "869b0a8575b54d08ae2817fad5dd42d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: UNKNOWN } } }
I20260925 03:03:56.303035 24664 tablet_bootstrap.cc:654] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:56.305938 24663 tablet_bootstrap.cc:492] T 11bb46bce3564841b7674e6a693bd2e6 P a04a37fc4aa7460cbfdbeeb4e451e1f5: No bootstrap required, opened a new log
I20260925 03:03:56.306526 24663 ts_tablet_manager.cc:1397] T 11bb46bce3564841b7674e6a693bd2e6 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent bootstrapping tablet: real 0.061s user 0.015s sys 0.007s
I20260925 03:03:56.308970 24663 raft_consensus.cc:348] T 11bb46bce3564841b7674e6a693bd2e6 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:56.309651 24663 raft_consensus.cc:374] T 11bb46bce3564841b7674e6a693bd2e6 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:56.309949 24663 raft_consensus.cc:729] T 11bb46bce3564841b7674e6a693bd2e6 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Initialized, Role: FOLLOWER
I20260925 03:03:56.310525 24663 consensus_queue.cc:260] T 11bb46bce3564841b7674e6a693bd2e6 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:56.312569 24663 ts_tablet_manager.cc:1428] T 11bb46bce3564841b7674e6a693bd2e6 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent starting tablet: real 0.006s user 0.002s sys 0.002s
I20260925 03:03:56.313486 24663 tablet_bootstrap.cc:492] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5: Bootstrap starting.
I20260925 03:03:56.315739 24662 tablet_bootstrap.cc:492] T 37a157996e944db2906110d87ecb0c54 P eeab93abfdb141a895002ec9f312296c: No bootstrap required, opened a new log
I20260925 03:03:56.319162 24663 tablet_bootstrap.cc:654] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:56.320191 24662 ts_tablet_manager.cc:1397] T 37a157996e944db2906110d87ecb0c54 P eeab93abfdb141a895002ec9f312296c: Time spent bootstrapping tablet: real 0.053s user 0.011s sys 0.009s
I20260925 03:03:56.322858 24662 raft_consensus.cc:348] T 37a157996e944db2906110d87ecb0c54 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:56.323617 24662 raft_consensus.cc:374] T 37a157996e944db2906110d87ecb0c54 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:56.323860 24662 raft_consensus.cc:729] T 37a157996e944db2906110d87ecb0c54 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Initialized, Role: FOLLOWER
I20260925 03:03:56.325191 24662 consensus_queue.cc:260] T 37a157996e944db2906110d87ecb0c54 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:56.327138 24662 ts_tablet_manager.cc:1428] T 37a157996e944db2906110d87ecb0c54 P eeab93abfdb141a895002ec9f312296c: Time spent starting tablet: real 0.007s user 0.003s sys 0.002s
I20260925 03:03:56.327286 24885 consensus_queue.cc:1035] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 03:03:56.328078 24662 tablet_bootstrap.cc:492] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c: Bootstrap starting.
I20260925 03:03:56.330420 24663 tablet_bootstrap.cc:492] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5: No bootstrap required, opened a new log
I20260925 03:03:56.330888 24663 ts_tablet_manager.cc:1397] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.013s
I20260925 03:03:56.333967 24663 raft_consensus.cc:348] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:56.334635 24663 raft_consensus.cc:374] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:56.334875 24663 raft_consensus.cc:729] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Initialized, Role: FOLLOWER
I20260925 03:03:56.335515 24663 consensus_queue.cc:260] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:56.338155 24663 ts_tablet_manager.cc:1428] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
I20260925 03:03:56.338333 24662 tablet_bootstrap.cc:654] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:56.339013 24663 tablet_bootstrap.cc:492] T cc2df40db08649c9a75e90000b878559 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Bootstrap starting.
I20260925 03:03:56.352180 24663 tablet_bootstrap.cc:654] T cc2df40db08649c9a75e90000b878559 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:56.357877 24708 consensus_queue.cc:1035] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260925 03:03:56.364974 24880 raft_consensus.cc:504] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:56.393316 24880 leader_election.cc:288] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:56.387495 24662 tablet_bootstrap.cc:492] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c: No bootstrap required, opened a new log
I20260925 03:03:56.384284 24663 tablet_bootstrap.cc:492] T cc2df40db08649c9a75e90000b878559 P a04a37fc4aa7460cbfdbeeb4e451e1f5: No bootstrap required, opened a new log
I20260925 03:03:56.394203 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:03:56.396397 24662 ts_tablet_manager.cc:1397] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c: Time spent bootstrapping tablet: real 0.069s user 0.010s sys 0.013s
I20260925 03:03:56.394349 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:03:56.396991 24619 raft_consensus.cc:3037] T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:56.397372 24664 tablet_bootstrap.cc:492] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5: No bootstrap required, opened a new log
I20260925 03:03:56.397922 24664 ts_tablet_manager.cc:1397] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5: Time spent bootstrapping tablet: real 0.159s user 0.007s sys 0.007s
I20260925 03:03:56.399082 24662 raft_consensus.cc:348] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:56.399785 24662 raft_consensus.cc:374] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:56.400055 24662 raft_consensus.cc:729] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Initialized, Role: FOLLOWER
I20260925 03:03:56.400580 24664 raft_consensus.cc:348] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:56.400801 24662 consensus_queue.cc:260] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:56.401335 24664 raft_consensus.cc:374] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:56.401763 24664 raft_consensus.cc:729] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Initialized, Role: FOLLOWER
I20260925 03:03:56.403007 24662 ts_tablet_manager.cc:1428] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260925 03:03:56.402604 24664 consensus_queue.cc:260] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:56.403927 24662 tablet_bootstrap.cc:492] T 550118bc1a0641ce9803bdbf43f2069f P eeab93abfdb141a895002ec9f312296c: Bootstrap starting.
I20260925 03:03:56.396415 24546 raft_consensus.cc:3037] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:56.395999 24663 ts_tablet_manager.cc:1397] T cc2df40db08649c9a75e90000b878559 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent bootstrapping tablet: real 0.057s user 0.010s sys 0.010s
I20260925 03:03:56.409086 24663 raft_consensus.cc:348] T cc2df40db08649c9a75e90000b878559 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:56.409578 24662 tablet_bootstrap.cc:654] T 550118bc1a0641ce9803bdbf43f2069f P eeab93abfdb141a895002ec9f312296c: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:56.411324 24664 ts_tablet_manager.cc:1428] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5: Time spent starting tablet: real 0.013s user 0.005s sys 0.000s
I20260925 03:03:56.412313 24664 tablet_bootstrap.cc:492] T c88cbd39e01f44628a094af516be2833 P 869b0a8575b54d08ae2817fad5dd42d5: Bootstrap starting.
I20260925 03:03:56.409639 24663 raft_consensus.cc:374] T cc2df40db08649c9a75e90000b878559 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:56.414757 24663 raft_consensus.cc:729] T cc2df40db08649c9a75e90000b878559 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Initialized, Role: FOLLOWER
I20260925 03:03:56.415413 24663 consensus_queue.cc:260] T cc2df40db08649c9a75e90000b878559 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:56.419481 24664 tablet_bootstrap.cc:654] T c88cbd39e01f44628a094af516be2833 P 869b0a8575b54d08ae2817fad5dd42d5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:56.420244 24546 raft_consensus.cc:2454] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 1.
I20260925 03:03:56.421547 24662 tablet_bootstrap.cc:492] T 550118bc1a0641ce9803bdbf43f2069f P eeab93abfdb141a895002ec9f312296c: No bootstrap required, opened a new log
I20260925 03:03:56.422035 24662 ts_tablet_manager.cc:1397] T 550118bc1a0641ce9803bdbf43f2069f P eeab93abfdb141a895002ec9f312296c: Time spent bootstrapping tablet: real 0.018s user 0.013s sys 0.003s
I20260925 03:03:56.422416 24663 ts_tablet_manager.cc:1428] T cc2df40db08649c9a75e90000b878559 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent starting tablet: real 0.015s user 0.005s sys 0.001s
I20260925 03:03:56.423393 24663 tablet_bootstrap.cc:492] T 8e5819c15eac4544a57e5ee6c9d4f117 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Bootstrap starting.
I20260925 03:03:56.424760 24619 raft_consensus.cc:2454] T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 1.
I20260925 03:03:56.437479 24437 leader_election.cc:302] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:03:56.438992 24885 raft_consensus.cc:482] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:56.439509 24885 raft_consensus.cc:504] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:56.441651 24885 leader_election.cc:288] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:56.442198 24885 raft_consensus.cc:2781] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:03:56.442977 24885 raft_consensus.cc:686] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 LEADER]: Becoming Leader. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:03:56.443706 24622 raft_consensus.cc:3037] T 37a157996e944db2906110d87ecb0c54 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:56.447779 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f1fac028a144058b92331c0d9b44ebf" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:03:56.448621 24621 raft_consensus.cc:2454] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 0.
I20260925 03:03:56.449904 24436 leader_election.cc:302] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:03:56.450831 24929 raft_consensus.cc:2781] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 03:03:56.451174 24929 raft_consensus.cc:482] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:03:56.451488 24929 raft_consensus.cc:3037] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:56.456519 24662 raft_consensus.cc:348] T 550118bc1a0641ce9803bdbf43f2069f P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:56.457316 24662 raft_consensus.cc:374] T 550118bc1a0641ce9803bdbf43f2069f P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:56.457588 24662 raft_consensus.cc:729] T 550118bc1a0641ce9803bdbf43f2069f P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Initialized, Role: FOLLOWER
I20260925 03:03:56.463037 24662 consensus_queue.cc:260] T 550118bc1a0641ce9803bdbf43f2069f P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:56.462801 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f1fac028a144058b92331c0d9b44ebf" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:03:56.443856 24885 consensus_queue.cc:237] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:56.470209 24662 ts_tablet_manager.cc:1428] T 550118bc1a0641ce9803bdbf43f2069f P eeab93abfdb141a895002ec9f312296c: Time spent starting tablet: real 0.048s user 0.008s sys 0.000s
I20260925 03:03:56.465107 24546 raft_consensus.cc:2454] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 0.
I20260925 03:03:56.471271 24662 tablet_bootstrap.cc:492] T 21c43af5e6844fdaacc8c625fa01fd8d P eeab93abfdb141a895002ec9f312296c: Bootstrap starting.
I20260925 03:03:56.472194 24663 tablet_bootstrap.cc:654] T 8e5819c15eac4544a57e5ee6c9d4f117 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:56.478685 24664 tablet_bootstrap.cc:492] T c88cbd39e01f44628a094af516be2833 P 869b0a8575b54d08ae2817fad5dd42d5: No bootstrap required, opened a new log
I20260925 03:03:56.479128 24664 ts_tablet_manager.cc:1397] T c88cbd39e01f44628a094af516be2833 P 869b0a8575b54d08ae2817fad5dd42d5: Time spent bootstrapping tablet: real 0.067s user 0.026s sys 0.009s
I20260925 03:03:56.483081 24664 raft_consensus.cc:348] T c88cbd39e01f44628a094af516be2833 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:56.483939 24664 raft_consensus.cc:374] T c88cbd39e01f44628a094af516be2833 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:56.484216 24664 raft_consensus.cc:729] T c88cbd39e01f44628a094af516be2833 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Initialized, Role: FOLLOWER
I20260925 03:03:56.484957 24664 consensus_queue.cc:260] T c88cbd39e01f44628a094af516be2833 P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:56.486467 24662 tablet_bootstrap.cc:654] T 21c43af5e6844fdaacc8c625fa01fd8d P eeab93abfdb141a895002ec9f312296c: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:56.491246 24664 ts_tablet_manager.cc:1428] T c88cbd39e01f44628a094af516be2833 P 869b0a8575b54d08ae2817fad5dd42d5: Time spent starting tablet: real 0.012s user 0.002s sys 0.005s
I20260925 03:03:56.492519 24664 tablet_bootstrap.cc:492] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5: Bootstrap starting.
I20260925 03:03:56.492216 24929 raft_consensus.cc:504] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:56.496127 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f1fac028a144058b92331c0d9b44ebf" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:03:56.496089 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f1fac028a144058b92331c0d9b44ebf" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:03:56.496836 24617 raft_consensus.cc:3037] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:56.496930 24546 raft_consensus.cc:3037] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:56.499033 24664 tablet_bootstrap.cc:654] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:56.503211 24617 raft_consensus.cc:2454] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 1.
I20260925 03:03:56.505749 24662 tablet_bootstrap.cc:492] T 21c43af5e6844fdaacc8c625fa01fd8d P eeab93abfdb141a895002ec9f312296c: No bootstrap required, opened a new log
I20260925 03:03:56.506223 24662 ts_tablet_manager.cc:1397] T 21c43af5e6844fdaacc8c625fa01fd8d P eeab93abfdb141a895002ec9f312296c: Time spent bootstrapping tablet: real 0.035s user 0.011s sys 0.000s
I20260925 03:03:56.506441 24436 leader_election.cc:302] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:03:56.507201 24708 raft_consensus.cc:2781] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:03:56.507642 24708 raft_consensus.cc:686] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 LEADER]: Becoming Leader. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:03:56.508406 24708 consensus_queue.cc:237] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:56.508512 24662 raft_consensus.cc:348] T 21c43af5e6844fdaacc8c625fa01fd8d P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:56.509248 24662 raft_consensus.cc:374] T 21c43af5e6844fdaacc8c625fa01fd8d P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:56.509508 24662 raft_consensus.cc:729] T 21c43af5e6844fdaacc8c625fa01fd8d P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Initialized, Role: FOLLOWER
I20260925 03:03:56.510143 24662 consensus_queue.cc:260] T 21c43af5e6844fdaacc8c625fa01fd8d P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:56.512246 24662 ts_tablet_manager.cc:1428] T 21c43af5e6844fdaacc8c625fa01fd8d P eeab93abfdb141a895002ec9f312296c: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260925 03:03:56.513533 24662 tablet_bootstrap.cc:492] T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c: Bootstrap starting.
I20260925 03:03:56.513145 24371 catalog_manager.cc:5495] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1). New cstate: current_term: 1 leader_uuid: "869b0a8575b54d08ae2817fad5dd42d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: UNKNOWN } } }
I20260925 03:03:56.510365 24546 raft_consensus.cc:2454] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 1.
I20260925 03:03:56.519762 24662 tablet_bootstrap.cc:654] T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:56.519940 24929 leader_election.cc:288] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:56.525373 24929 raft_consensus.cc:482] T c88cbd39e01f44628a094af516be2833 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:56.527355 24617 raft_consensus.cc:3037] T 550118bc1a0641ce9803bdbf43f2069f P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:56.527640 24929 raft_consensus.cc:504] T c88cbd39e01f44628a094af516be2833 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:56.529640 24663 tablet_bootstrap.cc:492] T 8e5819c15eac4544a57e5ee6c9d4f117 P a04a37fc4aa7460cbfdbeeb4e451e1f5: No bootstrap required, opened a new log
I20260925 03:03:56.530134 24663 ts_tablet_manager.cc:1397] T 8e5819c15eac4544a57e5ee6c9d4f117 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent bootstrapping tablet: real 0.107s user 0.011s sys 0.017s
I20260925 03:03:56.532068 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c88cbd39e01f44628a094af516be2833" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:03:56.529968 24929 leader_election.cc:288] T c88cbd39e01f44628a094af516be2833 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:56.533406 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c88cbd39e01f44628a094af516be2833" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
W20260925 03:03:56.534266 24437 leader_election.cc:341] T c88cbd39e01f44628a094af516be2833 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389): Illegal state: must be running to vote when last-logged opid is not known
W20260925 03:03:56.534873 24436 leader_election.cc:341] T c88cbd39e01f44628a094af516be2833 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:56.535281 24436 leader_election.cc:302] T c88cbd39e01f44628a094af516be2833 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:03:56.535989 24929 raft_consensus.cc:2726] T c88cbd39e01f44628a094af516be2833 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 03:03:56.537168 24929 consensus_queue.cc:1035] T 550118bc1a0641ce9803bdbf43f2069f P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 03:03:56.539551 24663 raft_consensus.cc:348] T 8e5819c15eac4544a57e5ee6c9d4f117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:56.540247 24663 raft_consensus.cc:374] T 8e5819c15eac4544a57e5ee6c9d4f117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:56.540521 24663 raft_consensus.cc:729] T 8e5819c15eac4544a57e5ee6c9d4f117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Initialized, Role: FOLLOWER
I20260925 03:03:56.541893 24663 consensus_queue.cc:260] T 8e5819c15eac4544a57e5ee6c9d4f117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:56.544328 24663 ts_tablet_manager.cc:1428] T 8e5819c15eac4544a57e5ee6c9d4f117 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent starting tablet: real 0.014s user 0.004s sys 0.003s
I20260925 03:03:56.544634 24664 tablet_bootstrap.cc:492] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5: No bootstrap required, opened a new log
I20260925 03:03:56.541697 24371 catalog_manager.cc:5495] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1). New cstate: current_term: 1 leader_uuid: "869b0a8575b54d08ae2817fad5dd42d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: UNKNOWN } } }
I20260925 03:03:56.549369 24664 ts_tablet_manager.cc:1397] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5: Time spent bootstrapping tablet: real 0.057s user 0.011s sys 0.005s
I20260925 03:03:56.549837 24663 tablet_bootstrap.cc:492] T 21c43af5e6844fdaacc8c625fa01fd8d P a04a37fc4aa7460cbfdbeeb4e451e1f5: Bootstrap starting.
I20260925 03:03:56.551496 24929 consensus_queue.cc:1035] T 550118bc1a0641ce9803bdbf43f2069f P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.010s
I20260925 03:03:56.565405 24664 raft_consensus.cc:348] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:56.566205 24664 raft_consensus.cc:374] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:56.566499 24664 raft_consensus.cc:729] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Initialized, Role: FOLLOWER
I20260925 03:03:56.567129 24664 consensus_queue.cc:260] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:56.572868 24664 ts_tablet_manager.cc:1428] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5: Time spent starting tablet: real 0.023s user 0.004s sys 0.003s
I20260925 03:03:56.573781 24664 tablet_bootstrap.cc:492] T ac9357d144aa49789faaf31fe5e6bf89 P 869b0a8575b54d08ae2817fad5dd42d5: Bootstrap starting.
I20260925 03:03:56.589424 24662 tablet_bootstrap.cc:492] T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c: No bootstrap required, opened a new log
I20260925 03:03:56.591701 24662 ts_tablet_manager.cc:1397] T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c: Time spent bootstrapping tablet: real 0.078s user 0.020s sys 0.009s
I20260925 03:03:56.593613 24664 tablet_bootstrap.cc:654] T ac9357d144aa49789faaf31fe5e6bf89 P 869b0a8575b54d08ae2817fad5dd42d5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:56.598165 24663 tablet_bootstrap.cc:654] T 21c43af5e6844fdaacc8c625fa01fd8d P a04a37fc4aa7460cbfdbeeb4e451e1f5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:56.600061 24662 raft_consensus.cc:348] T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:56.601598 24662 raft_consensus.cc:374] T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:56.601904 24662 raft_consensus.cc:729] T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Initialized, Role: FOLLOWER
I20260925 03:03:56.602854 24662 consensus_queue.cc:260] T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:56.604401 24662 ts_tablet_manager.cc:1428] T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c: Time spent starting tablet: real 0.012s user 0.005s sys 0.000s
I20260925 03:03:56.605314 24662 tablet_bootstrap.cc:492] T 15861cd28e354598a3c6bfc4337c34bb P eeab93abfdb141a895002ec9f312296c: Bootstrap starting.
I20260925 03:03:56.624491 24662 tablet_bootstrap.cc:654] T 15861cd28e354598a3c6bfc4337c34bb P eeab93abfdb141a895002ec9f312296c: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:56.631175 24879 consensus_queue.cc:1035] T 1414dcfe53f64a78ae413531eb8ec117 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260925 03:03:56.663110 24663 tablet_bootstrap.cc:492] T 21c43af5e6844fdaacc8c625fa01fd8d P a04a37fc4aa7460cbfdbeeb4e451e1f5: No bootstrap required, opened a new log
I20260925 03:03:56.663589 24663 ts_tablet_manager.cc:1397] T 21c43af5e6844fdaacc8c625fa01fd8d P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent bootstrapping tablet: real 0.118s user 0.019s sys 0.007s
I20260925 03:03:56.666060 24663 raft_consensus.cc:348] T 21c43af5e6844fdaacc8c625fa01fd8d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:56.666760 24663 raft_consensus.cc:374] T 21c43af5e6844fdaacc8c625fa01fd8d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:56.667021 24663 raft_consensus.cc:729] T 21c43af5e6844fdaacc8c625fa01fd8d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Initialized, Role: FOLLOWER
I20260925 03:03:56.667598 24663 consensus_queue.cc:260] T 21c43af5e6844fdaacc8c625fa01fd8d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:56.668426 24814 raft_consensus.cc:482] T 8e5819c15eac4544a57e5ee6c9d4f117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:56.668915 24814 raft_consensus.cc:504] T 8e5819c15eac4544a57e5ee6c9d4f117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:56.670814 24814 leader_election.cc:288] T 8e5819c15eac4544a57e5ee6c9d4f117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:56.671926 24663 ts_tablet_manager.cc:1428] T 21c43af5e6844fdaacc8c625fa01fd8d P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent starting tablet: real 0.008s user 0.007s sys 0.002s
I20260925 03:03:56.672266 24664 tablet_bootstrap.cc:492] T ac9357d144aa49789faaf31fe5e6bf89 P 869b0a8575b54d08ae2817fad5dd42d5: No bootstrap required, opened a new log
I20260925 03:03:56.672099 24885 consensus_queue.cc:1035] T 1414dcfe53f64a78ae413531eb8ec117 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 03:03:56.672729 24664 ts_tablet_manager.cc:1397] T ac9357d144aa49789faaf31fe5e6bf89 P 869b0a8575b54d08ae2817fad5dd42d5: Time spent bootstrapping tablet: real 0.099s user 0.015s sys 0.013s
I20260925 03:03:56.672894 24662 tablet_bootstrap.cc:492] T 15861cd28e354598a3c6bfc4337c34bb P eeab93abfdb141a895002ec9f312296c: No bootstrap required, opened a new log
I20260925 03:03:56.672931 24663 tablet_bootstrap.cc:492] T 6c81a1b3b616492ab6c964f45af36077 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Bootstrap starting.
I20260925 03:03:56.673429 24662 ts_tablet_manager.cc:1397] T 15861cd28e354598a3c6bfc4337c34bb P eeab93abfdb141a895002ec9f312296c: Time spent bootstrapping tablet: real 0.068s user 0.013s sys 0.014s
I20260925 03:03:56.675439 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e5819c15eac4544a57e5ee6c9d4f117" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:03:56.675885 24662 raft_consensus.cc:348] T 15861cd28e354598a3c6bfc4337c34bb P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:56.676559 24662 raft_consensus.cc:374] T 15861cd28e354598a3c6bfc4337c34bb P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:56.676849 24662 raft_consensus.cc:729] T 15861cd28e354598a3c6bfc4337c34bb P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Initialized, Role: FOLLOWER
I20260925 03:03:56.675288 24664 raft_consensus.cc:348] T ac9357d144aa49789faaf31fe5e6bf89 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:56.677590 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e5819c15eac4544a57e5ee6c9d4f117" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:03:56.677434 24662 consensus_queue.cc:260] T 15861cd28e354598a3c6bfc4337c34bb P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:56.678025 24664 raft_consensus.cc:374] T ac9357d144aa49789faaf31fe5e6bf89 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:56.678303 24664 raft_consensus.cc:729] T ac9357d144aa49789faaf31fe5e6bf89 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Initialized, Role: FOLLOWER
I20260925 03:03:56.678891 24664 consensus_queue.cc:260] T ac9357d144aa49789faaf31fe5e6bf89 P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:03:56.679293 24511 leader_election.cc:341] T 8e5819c15eac4544a57e5ee6c9d4f117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105): Illegal state: must be running to vote when last-logged opid is not known
W20260925 03:03:56.680667 24514 leader_election.cc:341] T 8e5819c15eac4544a57e5ee6c9d4f117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:56.681105 24514 leader_election.cc:302] T 8e5819c15eac4544a57e5ee6c9d4f117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:03:56.682003 24664 ts_tablet_manager.cc:1428] T ac9357d144aa49789faaf31fe5e6bf89 P 869b0a8575b54d08ae2817fad5dd42d5: Time spent starting tablet: real 0.009s user 0.000s sys 0.005s
I20260925 03:03:56.682199 24814 raft_consensus.cc:2726] T 8e5819c15eac4544a57e5ee6c9d4f117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 03:03:56.682556 24663 tablet_bootstrap.cc:654] T 6c81a1b3b616492ab6c964f45af36077 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:56.682900 24664 tablet_bootstrap.cc:492] T 0be4e0aa984f4f2b81a58e382e434ad5 P 869b0a8575b54d08ae2817fad5dd42d5: Bootstrap starting.
I20260925 03:03:56.687330 24814 raft_consensus.cc:482] T cc2df40db08649c9a75e90000b878559 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:56.687907 24814 raft_consensus.cc:504] T cc2df40db08649c9a75e90000b878559 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:56.689877 24664 tablet_bootstrap.cc:654] T 0be4e0aa984f4f2b81a58e382e434ad5 P 869b0a8575b54d08ae2817fad5dd42d5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:56.690184 24885 raft_consensus.cc:482] T 21c43af5e6844fdaacc8c625fa01fd8d P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:56.690677 24885 raft_consensus.cc:504] T 21c43af5e6844fdaacc8c625fa01fd8d P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:56.692786 24662 ts_tablet_manager.cc:1428] T 15861cd28e354598a3c6bfc4337c34bb P eeab93abfdb141a895002ec9f312296c: Time spent starting tablet: real 0.019s user 0.006s sys 0.000s
I20260925 03:03:56.693063 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc2df40db08649c9a75e90000b878559" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:03:56.693795 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21c43af5e6844fdaacc8c625fa01fd8d" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:03:56.694396 24546 raft_consensus.cc:2454] T 21c43af5e6844fdaacc8c625fa01fd8d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 0.
I20260925 03:03:56.695312 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21c43af5e6844fdaacc8c625fa01fd8d" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:03:56.695683 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc2df40db08649c9a75e90000b878559" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:03:56.695956 24617 raft_consensus.cc:2454] T 21c43af5e6844fdaacc8c625fa01fd8d P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 0.
W20260925 03:03:56.697147 24511 leader_election.cc:341] T cc2df40db08649c9a75e90000b878559 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:56.697193 24436 leader_election.cc:302] T 21c43af5e6844fdaacc8c625fa01fd8d P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:03:56.697716 24885 leader_election.cc:288] T 21c43af5e6844fdaacc8c625fa01fd8d P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:56.698114 24929 raft_consensus.cc:2781] T 21c43af5e6844fdaacc8c625fa01fd8d P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 03:03:56.698436 24662 tablet_bootstrap.cc:492] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c: Bootstrap starting.
I20260925 03:03:56.698524 24929 raft_consensus.cc:482] T 21c43af5e6844fdaacc8c625fa01fd8d P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:03:56.699283 24929 raft_consensus.cc:3037] T 21c43af5e6844fdaacc8c625fa01fd8d P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:56.702129 24663 tablet_bootstrap.cc:492] T 6c81a1b3b616492ab6c964f45af36077 P a04a37fc4aa7460cbfdbeeb4e451e1f5: No bootstrap required, opened a new log
I20260925 03:03:56.702381 24664 tablet_bootstrap.cc:492] T 0be4e0aa984f4f2b81a58e382e434ad5 P 869b0a8575b54d08ae2817fad5dd42d5: No bootstrap required, opened a new log
I20260925 03:03:56.702704 24663 ts_tablet_manager.cc:1397] T 6c81a1b3b616492ab6c964f45af36077 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent bootstrapping tablet: real 0.030s user 0.013s sys 0.004s
I20260925 03:03:56.702957 24664 ts_tablet_manager.cc:1397] T 0be4e0aa984f4f2b81a58e382e434ad5 P 869b0a8575b54d08ae2817fad5dd42d5: Time spent bootstrapping tablet: real 0.020s user 0.008s sys 0.008s
I20260925 03:03:56.705552 24662 tablet_bootstrap.cc:654] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:56.705634 24663 raft_consensus.cc:348] T 6c81a1b3b616492ab6c964f45af36077 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:56.705967 24664 raft_consensus.cc:348] T 0be4e0aa984f4f2b81a58e382e434ad5 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:56.706398 24663 raft_consensus.cc:374] T 6c81a1b3b616492ab6c964f45af36077 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:56.706744 24663 raft_consensus.cc:729] T 6c81a1b3b616492ab6c964f45af36077 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Initialized, Role: FOLLOWER
I20260925 03:03:56.707567 24663 consensus_queue.cc:260] T 6c81a1b3b616492ab6c964f45af36077 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:56.711728 24663 ts_tablet_manager.cc:1428] T 6c81a1b3b616492ab6c964f45af36077 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260925 03:03:56.712491 24885 consensus_queue.cc:1035] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 03:03:56.690193 24814 leader_election.cc:288] T cc2df40db08649c9a75e90000b878559 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
W20260925 03:03:56.712742 24514 leader_election.cc:341] T cc2df40db08649c9a75e90000b878559 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:56.706710 24664 raft_consensus.cc:374] T 0be4e0aa984f4f2b81a58e382e434ad5 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:56.713887 24514 leader_election.cc:302] T cc2df40db08649c9a75e90000b878559 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:03:56.714078 24664 raft_consensus.cc:729] T 0be4e0aa984f4f2b81a58e382e434ad5 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Initialized, Role: FOLLOWER
I20260925 03:03:56.714975 24814 raft_consensus.cc:2726] T cc2df40db08649c9a75e90000b878559 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 03:03:56.715710 24663 tablet_bootstrap.cc:492] T c88cbd39e01f44628a094af516be2833 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Bootstrap starting.
I20260925 03:03:56.715994 24664 consensus_queue.cc:260] T 0be4e0aa984f4f2b81a58e382e434ad5 P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:56.718710 24664 ts_tablet_manager.cc:1428] T 0be4e0aa984f4f2b81a58e382e434ad5 P 869b0a8575b54d08ae2817fad5dd42d5: Time spent starting tablet: real 0.015s user 0.008s sys 0.000s
I20260925 03:03:56.719628 24664 tablet_bootstrap.cc:492] T 11bb46bce3564841b7674e6a693bd2e6 P 869b0a8575b54d08ae2817fad5dd42d5: Bootstrap starting.
I20260925 03:03:56.720302 24885 consensus_queue.cc:1035] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 03:03:56.721915 24663 tablet_bootstrap.cc:654] T c88cbd39e01f44628a094af516be2833 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:56.725215 24664 tablet_bootstrap.cc:654] T 11bb46bce3564841b7674e6a693bd2e6 P 869b0a8575b54d08ae2817fad5dd42d5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:56.726681 24929 raft_consensus.cc:504] T 21c43af5e6844fdaacc8c625fa01fd8d P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:56.752808 24929 leader_election.cc:288] T 21c43af5e6844fdaacc8c625fa01fd8d P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:56.753206 24662 tablet_bootstrap.cc:492] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c: No bootstrap required, opened a new log
I20260925 03:03:56.763491 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21c43af5e6844fdaacc8c625fa01fd8d" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:03:56.764171 24546 raft_consensus.cc:3037] T 21c43af5e6844fdaacc8c625fa01fd8d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:56.782128 24662 ts_tablet_manager.cc:1397] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c: Time spent bootstrapping tablet: real 0.084s user 0.022s sys 0.002s
I20260925 03:03:56.787259 24546 raft_consensus.cc:2454] T 21c43af5e6844fdaacc8c625fa01fd8d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 1.
I20260925 03:03:56.787771 24662 raft_consensus.cc:348] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:56.789788 24931 raft_consensus.cc:482] T 11bb46bce3564841b7674e6a693bd2e6 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:56.793784 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21c43af5e6844fdaacc8c625fa01fd8d" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:03:56.801050 24619 raft_consensus.cc:3037] T 21c43af5e6844fdaacc8c625fa01fd8d P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:56.801750 24931 raft_consensus.cc:504] T 11bb46bce3564841b7674e6a693bd2e6 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:56.796470 24663 tablet_bootstrap.cc:492] T c88cbd39e01f44628a094af516be2833 P a04a37fc4aa7460cbfdbeeb4e451e1f5: No bootstrap required, opened a new log
I20260925 03:03:56.798425 24662 raft_consensus.cc:374] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:56.803740 24931 leader_election.cc:288] T 11bb46bce3564841b7674e6a693bd2e6 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:56.803822 24663 ts_tablet_manager.cc:1397] T c88cbd39e01f44628a094af516be2833 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent bootstrapping tablet: real 0.088s user 0.010s sys 0.011s
I20260925 03:03:56.803857 24662 raft_consensus.cc:729] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Initialized, Role: FOLLOWER
I20260925 03:03:56.812028 24662 consensus_queue.cc:260] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:56.816965 24617 raft_consensus.cc:3037] T 15861cd28e354598a3c6bfc4337c34bb P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:56.817603 24663 raft_consensus.cc:348] T c88cbd39e01f44628a094af516be2833 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:56.818271 24663 raft_consensus.cc:374] T c88cbd39e01f44628a094af516be2833 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:56.818516 24663 raft_consensus.cc:729] T c88cbd39e01f44628a094af516be2833 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Initialized, Role: FOLLOWER
I20260925 03:03:56.819090 24663 consensus_queue.cc:260] T c88cbd39e01f44628a094af516be2833 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:56.821710 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bb46bce3564841b7674e6a693bd2e6" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
W20260925 03:03:56.824352 24511 leader_election.cc:341] T 11bb46bce3564841b7674e6a693bd2e6 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:56.824522 24437 leader_election.cc:302] T 21c43af5e6844fdaacc8c625fa01fd8d P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:03:56.827072 24663 ts_tablet_manager.cc:1428] T c88cbd39e01f44628a094af516be2833 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent starting tablet: real 0.023s user 0.003s sys 0.002s
I20260925 03:03:56.827944 24663 tablet_bootstrap.cc:492] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5: Bootstrap starting.
I20260925 03:03:56.835829 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bb46bce3564841b7674e6a693bd2e6" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:03:56.836263 24662 ts_tablet_manager.cc:1428] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c: Time spent starting tablet: real 0.054s user 0.007s sys 0.000s
I20260925 03:03:56.837203 24662 tablet_bootstrap.cc:492] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c: Bootstrap starting.
I20260925 03:03:56.838310 24664 tablet_bootstrap.cc:492] T 11bb46bce3564841b7674e6a693bd2e6 P 869b0a8575b54d08ae2817fad5dd42d5: No bootstrap required, opened a new log
I20260925 03:03:56.838701 24664 ts_tablet_manager.cc:1397] T 11bb46bce3564841b7674e6a693bd2e6 P 869b0a8575b54d08ae2817fad5dd42d5: Time spent bootstrapping tablet: real 0.119s user 0.010s sys 0.007s
I20260925 03:03:56.830715 24708 raft_consensus.cc:2781] T 21c43af5e6844fdaacc8c625fa01fd8d P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:03:56.841560 24708 raft_consensus.cc:686] T 21c43af5e6844fdaacc8c625fa01fd8d P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 LEADER]: Becoming Leader. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:03:56.843231 24708 consensus_queue.cc:237] T 21c43af5e6844fdaacc8c625fa01fd8d P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:56.831837 24870 raft_consensus.cc:482] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:56.845698 24870 raft_consensus.cc:504] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:56.832875 24885 raft_consensus.cc:482] T 0be4e0aa984f4f2b81a58e382e434ad5 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:56.847084 24664 raft_consensus.cc:348] T 11bb46bce3564841b7674e6a693bd2e6 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:56.847527 24885 raft_consensus.cc:504] T 0be4e0aa984f4f2b81a58e382e434ad5 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:56.847934 24664 raft_consensus.cc:374] T 11bb46bce3564841b7674e6a693bd2e6 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:56.848325 24664 raft_consensus.cc:729] T 11bb46bce3564841b7674e6a693bd2e6 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Initialized, Role: FOLLOWER
I20260925 03:03:56.848996 24664 consensus_queue.cc:260] T 11bb46bce3564841b7674e6a693bd2e6 P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:56.851121 24664 ts_tablet_manager.cc:1428] T 11bb46bce3564841b7674e6a693bd2e6 P 869b0a8575b54d08ae2817fad5dd42d5: Time spent starting tablet: real 0.012s user 0.005s sys 0.001s
I20260925 03:03:56.852351 24664 tablet_bootstrap.cc:492] T 8e5819c15eac4544a57e5ee6c9d4f117 P 869b0a8575b54d08ae2817fad5dd42d5: Bootstrap starting.
W20260925 03:03:56.858139 24514 leader_election.cc:341] T 11bb46bce3564841b7674e6a693bd2e6 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:56.858605 24514 leader_election.cc:302] T 11bb46bce3564841b7674e6a693bd2e6 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:03:56.861714 24870 leader_election.cc:288] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:56.861937 24921 raft_consensus.cc:2726] T 11bb46bce3564841b7674e6a693bd2e6 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 03:03:56.863929 24885 leader_election.cc:288] T 0be4e0aa984f4f2b81a58e382e434ad5 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:56.868094 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be4e0aa984f4f2b81a58e382e434ad5" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:03:56.868876 24621 raft_consensus.cc:2454] T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 0.
I20260925 03:03:56.870041 24885 consensus_queue.cc:1035] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 03:03:56.870384 24879 consensus_queue.cc:1035] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260925 03:03:56.873056 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceab6b0fccd54e909a64f66fe81a9a1f" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:03:56.874377 24436 leader_election.cc:302] T 0be4e0aa984f4f2b81a58e382e434ad5 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:03:56.875427 24663 tablet_bootstrap.cc:654] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:56.875957 24879 raft_consensus.cc:2781] T 0be4e0aa984f4f2b81a58e382e434ad5 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 03:03:56.876351 24879 raft_consensus.cc:482] T 0be4e0aa984f4f2b81a58e382e434ad5 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:03:56.876912 24879 raft_consensus.cc:3037] T 0be4e0aa984f4f2b81a58e382e434ad5 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:56.880036 24664 tablet_bootstrap.cc:654] T 8e5819c15eac4544a57e5ee6c9d4f117 P 869b0a8575b54d08ae2817fad5dd42d5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:56.881713 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be4e0aa984f4f2b81a58e382e434ad5" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:03:56.882336 24619 raft_consensus.cc:2454] T 21c43af5e6844fdaacc8c625fa01fd8d P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 1.
I20260925 03:03:56.882439 24547 raft_consensus.cc:2454] T 0be4e0aa984f4f2b81a58e382e434ad5 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 0.
W20260925 03:03:56.883973 24511 leader_election.cc:341] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:56.894382 24880 raft_consensus.cc:482] T 11bb46bce3564841b7674e6a693bd2e6 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:56.895005 24880 raft_consensus.cc:504] T 11bb46bce3564841b7674e6a693bd2e6 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:56.896291 24663 tablet_bootstrap.cc:492] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5: No bootstrap required, opened a new log
I20260925 03:03:56.896764 24663 ts_tablet_manager.cc:1397] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent bootstrapping tablet: real 0.069s user 0.006s sys 0.005s
I20260925 03:03:56.897009 24880 leader_election.cc:288] T 11bb46bce3564841b7674e6a693bd2e6 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:56.899019 24663 raft_consensus.cc:348] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:56.899727 24663 raft_consensus.cc:374] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:56.899981 24663 raft_consensus.cc:729] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Initialized, Role: FOLLOWER
I20260925 03:03:56.903106 24663 consensus_queue.cc:260] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:56.906021 24663 ts_tablet_manager.cc:1428] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent starting tablet: real 0.009s user 0.004s sys 0.001s
I20260925 03:03:56.906966 24663 tablet_bootstrap.cc:492] T 25102ba08e4e49388f6176badcd10a25 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Bootstrap starting.
I20260925 03:03:56.899940 24662 tablet_bootstrap.cc:654] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:56.915118 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceab6b0fccd54e909a64f66fe81a9a1f" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:03:56.917080 24664 tablet_bootstrap.cc:492] T 8e5819c15eac4544a57e5ee6c9d4f117 P 869b0a8575b54d08ae2817fad5dd42d5: No bootstrap required, opened a new log
I20260925 03:03:56.917618 24664 ts_tablet_manager.cc:1397] T 8e5819c15eac4544a57e5ee6c9d4f117 P 869b0a8575b54d08ae2817fad5dd42d5: Time spent bootstrapping tablet: real 0.065s user 0.006s sys 0.008s
W20260925 03:03:56.925020 24514 leader_election.cc:341] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:56.925487 24514 leader_election.cc:302] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:03:56.926160 24814 raft_consensus.cc:2726] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 03:03:56.927266 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bb46bce3564841b7674e6a693bd2e6" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:03:56.927891 24546 raft_consensus.cc:2454] T 11bb46bce3564841b7674e6a693bd2e6 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 0.
I20260925 03:03:56.928419 24371 catalog_manager.cc:5495] T 21c43af5e6844fdaacc8c625fa01fd8d P 869b0a8575b54d08ae2817fad5dd42d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1). New cstate: current_term: 1 leader_uuid: "869b0a8575b54d08ae2817fad5dd42d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: UNKNOWN } } }
I20260925 03:03:56.934769 24880 raft_consensus.cc:482] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:56.935266 24880 raft_consensus.cc:504] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:56.936568 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bb46bce3564841b7674e6a693bd2e6" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:03:56.936371 24664 raft_consensus.cc:348] T 8e5819c15eac4544a57e5ee6c9d4f117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:56.937161 24664 raft_consensus.cc:374] T 8e5819c15eac4544a57e5ee6c9d4f117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:56.937448 24664 raft_consensus.cc:729] T 8e5819c15eac4544a57e5ee6c9d4f117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Initialized, Role: FOLLOWER
I20260925 03:03:56.938112 24664 consensus_queue.cc:260] T 8e5819c15eac4544a57e5ee6c9d4f117 P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:56.946171 24664 ts_tablet_manager.cc:1428] T 8e5819c15eac4544a57e5ee6c9d4f117 P 869b0a8575b54d08ae2817fad5dd42d5: Time spent starting tablet: real 0.028s user 0.005s sys 0.001s
I20260925 03:03:56.949819 24879 raft_consensus.cc:504] T 0be4e0aa984f4f2b81a58e382e434ad5 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:56.950425 24663 tablet_bootstrap.cc:654] T 25102ba08e4e49388f6176badcd10a25 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:56.953143 24437 leader_election.cc:302] T 11bb46bce3564841b7674e6a693bd2e6 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:03:56.953258 24664 tablet_bootstrap.cc:492] T 25102ba08e4e49388f6176badcd10a25 P 869b0a8575b54d08ae2817fad5dd42d5: Bootstrap starting.
I20260925 03:03:56.952787 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9c33fd100ff44c1a3247319a324a044" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
W20260925 03:03:56.958380 24436 leader_election.cc:341] T 11bb46bce3564841b7674e6a693bd2e6 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:56.937115 24880 leader_election.cc:288] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:56.956120 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9c33fd100ff44c1a3247319a324a044" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:03:56.959205 24929 raft_consensus.cc:2781] T 11bb46bce3564841b7674e6a693bd2e6 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 03:03:56.959856 24929 raft_consensus.cc:482] T 11bb46bce3564841b7674e6a693bd2e6 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:03:56.960556 24929 raft_consensus.cc:3037] T 11bb46bce3564841b7674e6a693bd2e6 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:56.960878 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be4e0aa984f4f2b81a58e382e434ad5" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:03:56.961499 24546 raft_consensus.cc:3037] T 0be4e0aa984f4f2b81a58e382e434ad5 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Advancing to term 1
W20260925 03:03:56.963977 24437 leader_election.cc:341] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:56.967777 24880 consensus_queue.cc:1035] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 03:03:56.972419 24929 raft_consensus.cc:504] T 11bb46bce3564841b7674e6a693bd2e6 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:56.983700 24546 raft_consensus.cc:2454] T 0be4e0aa984f4f2b81a58e382e434ad5 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 1.
I20260925 03:03:56.984022 24880 raft_consensus.cc:482] T ac9357d144aa49789faaf31fe5e6bf89 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:56.985354 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bb46bce3564841b7674e6a693bd2e6" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:03:56.985208 24880 raft_consensus.cc:504] T ac9357d144aa49789faaf31fe5e6bf89 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:56.985952 24544 raft_consensus.cc:3037] T 11bb46bce3564841b7674e6a693bd2e6 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:56.987205 24880 leader_election.cc:288] T ac9357d144aa49789faaf31fe5e6bf89 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
W20260925 03:03:56.988164 24418 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260925 03:03:56.988588 24437 leader_election.cc:302] T 0be4e0aa984f4f2b81a58e382e434ad5 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:03:56.981540 24929 leader_election.cc:288] T 11bb46bce3564841b7674e6a693bd2e6 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:56.991155 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9357d144aa49789faaf31fe5e6bf89" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:03:56.991817 24546 raft_consensus.cc:2454] T ac9357d144aa49789faaf31fe5e6bf89 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 0.
I20260925 03:03:56.990571 24880 raft_consensus.cc:2781] T 0be4e0aa984f4f2b81a58e382e434ad5 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:03:56.992924 24437 leader_election.cc:302] T ac9357d144aa49789faaf31fe5e6bf89 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:03:56.998338 24544 raft_consensus.cc:2454] T 11bb46bce3564841b7674e6a693bd2e6 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 1.
I20260925 03:03:56.999369 24880 raft_consensus.cc:686] T 0be4e0aa984f4f2b81a58e382e434ad5 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 LEADER]: Becoming Leader. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:03:56.989797 24879 leader_election.cc:288] T 0be4e0aa984f4f2b81a58e382e434ad5 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:57.000087 24708 consensus_queue.cc:1035] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 03:03:57.000391 24880 consensus_queue.cc:237] T 0be4e0aa984f4f2b81a58e382e434ad5 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:57.004832 24662 tablet_bootstrap.cc:492] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c: No bootstrap required, opened a new log
I20260925 03:03:57.004985 24885 raft_consensus.cc:2781] T ac9357d144aa49789faaf31fe5e6bf89 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 03:03:57.005297 24662 ts_tablet_manager.cc:1397] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c: Time spent bootstrapping tablet: real 0.168s user 0.017s sys 0.005s
I20260925 03:03:57.005409 24885 raft_consensus.cc:482] T ac9357d144aa49789faaf31fe5e6bf89 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:03:57.005846 24885 raft_consensus.cc:3037] T ac9357d144aa49789faaf31fe5e6bf89 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:57.007931 24662 raft_consensus.cc:348] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:57.008663 24662 raft_consensus.cc:374] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:57.009121 24662 raft_consensus.cc:729] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Initialized, Role: FOLLOWER
I20260925 03:03:57.009922 24662 consensus_queue.cc:260] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:57.010272 24663 tablet_bootstrap.cc:492] T 25102ba08e4e49388f6176badcd10a25 P a04a37fc4aa7460cbfdbeeb4e451e1f5: No bootstrap required, opened a new log
I20260925 03:03:57.011154 24663 ts_tablet_manager.cc:1397] T 25102ba08e4e49388f6176badcd10a25 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent bootstrapping tablet: real 0.104s user 0.006s sys 0.019s
I20260925 03:03:57.012089 24662 ts_tablet_manager.cc:1428] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c: Time spent starting tablet: real 0.006s user 0.001s sys 0.004s
I20260925 03:03:57.010743 24664 tablet_bootstrap.cc:654] T 25102ba08e4e49388f6176badcd10a25 P 869b0a8575b54d08ae2817fad5dd42d5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:57.012621 24885 raft_consensus.cc:504] T ac9357d144aa49789faaf31fe5e6bf89 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:57.013999 24663 raft_consensus.cc:348] T 25102ba08e4e49388f6176badcd10a25 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:57.014093 24662 tablet_bootstrap.cc:492] T 25102ba08e4e49388f6176badcd10a25 P eeab93abfdb141a895002ec9f312296c: Bootstrap starting.
I20260925 03:03:57.014590 24885 leader_election.cc:288] T ac9357d144aa49789faaf31fe5e6bf89 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:57.014755 24663 raft_consensus.cc:374] T 25102ba08e4e49388f6176badcd10a25 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:57.015252 24663 raft_consensus.cc:729] T 25102ba08e4e49388f6176badcd10a25 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Initialized, Role: FOLLOWER
I20260925 03:03:57.015727 24663 consensus_queue.cc:260] T 25102ba08e4e49388f6176badcd10a25 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:57.016144 24437 leader_election.cc:302] T 11bb46bce3564841b7674e6a693bd2e6 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
W20260925 03:03:57.018311 24436 leader_election.cc:341] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:57.018747 24436 leader_election.cc:302] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:03:57.021307 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bb46bce3564841b7674e6a693bd2e6" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:03:57.021727 24880 raft_consensus.cc:2726] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 03:03:57.021310 24618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9357d144aa49789faaf31fe5e6bf89" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:03:57.022418 24879 raft_consensus.cc:2781] T 11bb46bce3564841b7674e6a693bd2e6 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:03:57.022919 24879 raft_consensus.cc:686] T 11bb46bce3564841b7674e6a693bd2e6 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 LEADER]: Becoming Leader. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:03:57.023691 24879 consensus_queue.cc:237] T 11bb46bce3564841b7674e6a693bd2e6 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:57.028378 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9357d144aa49789faaf31fe5e6bf89" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:03:57.030229 24662 tablet_bootstrap.cc:654] T 25102ba08e4e49388f6176badcd10a25 P eeab93abfdb141a895002ec9f312296c: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:57.031196 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be4e0aa984f4f2b81a58e382e434ad5" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:03:57.031422 24928 raft_consensus.cc:482] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:57.033838 24620 raft_consensus.cc:3037] T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:57.033892 24928 raft_consensus.cc:504] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:57.034837 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9357d144aa49789faaf31fe5e6bf89" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:03:57.035462 24547 raft_consensus.cc:3037] T ac9357d144aa49789faaf31fe5e6bf89 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:57.035966 24928 leader_election.cc:288] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:03:57.037602 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e5819c15eac4544a57e5ee6c9d4f117" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:03:57.038298 24471 raft_consensus.cc:2454] T 8e5819c15eac4544a57e5ee6c9d4f117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 0.
I20260925 03:03:57.039131 24664 tablet_bootstrap.cc:492] T 25102ba08e4e49388f6176badcd10a25 P 869b0a8575b54d08ae2817fad5dd42d5: No bootstrap required, opened a new log
I20260925 03:03:57.039417 24588 leader_election.cc:302] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:03:57.039626 24664 ts_tablet_manager.cc:1397] T 25102ba08e4e49388f6176badcd10a25 P 869b0a8575b54d08ae2817fad5dd42d5: Time spent bootstrapping tablet: real 0.088s user 0.007s sys 0.007s
I20260925 03:03:57.041074 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e5819c15eac4544a57e5ee6c9d4f117" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:03:57.041754 24544 raft_consensus.cc:2454] T 8e5819c15eac4544a57e5ee6c9d4f117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 0.
I20260925 03:03:57.041869 24928 raft_consensus.cc:2781] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 03:03:57.042896 24928 raft_consensus.cc:482] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:03:57.042397 24664 raft_consensus.cc:348] T 25102ba08e4e49388f6176badcd10a25 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:57.043479 24664 raft_consensus.cc:374] T 25102ba08e4e49388f6176badcd10a25 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:57.043558 24663 ts_tablet_manager.cc:1428] T 25102ba08e4e49388f6176badcd10a25 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent starting tablet: real 0.032s user 0.005s sys 0.001s
I20260925 03:03:57.043262 24928 raft_consensus.cc:3037] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:57.043882 24664 raft_consensus.cc:729] T 25102ba08e4e49388f6176badcd10a25 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Initialized, Role: FOLLOWER
I20260925 03:03:57.044706 24663 tablet_bootstrap.cc:492] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Bootstrap starting.
I20260925 03:03:57.044857 24664 consensus_queue.cc:260] T 25102ba08e4e49388f6176badcd10a25 P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:57.047114 24664 ts_tablet_manager.cc:1428] T 25102ba08e4e49388f6176badcd10a25 P 869b0a8575b54d08ae2817fad5dd42d5: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260925 03:03:57.049355 24664 tablet_bootstrap.cc:492] T e6b6ec010caf4689b8c90449764f5cf9 P 869b0a8575b54d08ae2817fad5dd42d5: Bootstrap starting.
W20260925 03:03:57.050218 24436 leader_election.cc:341] T 11bb46bce3564841b7674e6a693bd2e6 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:57.041922 24620 raft_consensus.cc:2454] T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 1.
I20260925 03:03:57.053150 24708 consensus_queue.cc:1035] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 03:03:57.055510 24436 leader_election.cc:341] T ac9357d144aa49789faaf31fe5e6bf89 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105): Illegal state: must be running to vote when last-logged opid is not known
W20260925 03:03:57.056146 24436 leader_election.cc:341] T ac9357d144aa49789faaf31fe5e6bf89 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:57.058267 24367 catalog_manager.cc:5495] T 0be4e0aa984f4f2b81a58e382e434ad5 P 869b0a8575b54d08ae2817fad5dd42d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1). New cstate: current_term: 1 leader_uuid: "869b0a8575b54d08ae2817fad5dd42d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: UNKNOWN } } }
I20260925 03:03:57.060086 24663 tablet_bootstrap.cc:654] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:57.078611 24921 raft_consensus.cc:482] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:57.085175 24664 tablet_bootstrap.cc:654] T e6b6ec010caf4689b8c90449764f5cf9 P 869b0a8575b54d08ae2817fad5dd42d5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:57.085476 24921 raft_consensus.cc:504] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:57.076625 24547 raft_consensus.cc:2454] T ac9357d144aa49789faaf31fe5e6bf89 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 1.
I20260925 03:03:57.079387 24885 consensus_queue.cc:1035] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.008s
I20260925 03:03:57.088204 24437 leader_election.cc:302] T ac9357d144aa49789faaf31fe5e6bf89 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:03:57.091738 24708 raft_consensus.cc:2781] T ac9357d144aa49789faaf31fe5e6bf89 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:03:57.092921 24708 raft_consensus.cc:686] T ac9357d144aa49789faaf31fe5e6bf89 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 LEADER]: Becoming Leader. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:03:57.094146 24931 raft_consensus.cc:482] T 6c81a1b3b616492ab6c964f45af36077 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:57.093763 24708 consensus_queue.cc:237] T ac9357d144aa49789faaf31fe5e6bf89 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:57.094599 24931 raft_consensus.cc:504] T 6c81a1b3b616492ab6c964f45af36077 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:57.098650 24931 leader_election.cc:288] T 6c81a1b3b616492ab6c964f45af36077 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:57.098753 24921 leader_election.cc:288] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:57.100720 24616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c81a1b3b616492ab6c964f45af36077" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:03:57.101722 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5f9b503b0ab43999d582015a367441e" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:03:57.102370 24622 raft_consensus.cc:2454] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 0.
I20260925 03:03:57.103440 24511 leader_election.cc:302] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:03:57.104871 24921 raft_consensus.cc:2781] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 03:03:57.105263 24921 raft_consensus.cc:482] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:03:57.105578 24921 raft_consensus.cc:3037] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Advancing to term 1
W20260925 03:03:57.107580 24511 leader_election.cc:341] T 6c81a1b3b616492ab6c964f45af36077 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:57.060523 24928 raft_consensus.cc:504] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:57.115473 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5f9b503b0ab43999d582015a367441e" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
W20260925 03:03:57.117528 24514 leader_election.cc:341] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:57.113688 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c81a1b3b616492ab6c964f45af36077" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:03:57.119207 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e5819c15eac4544a57e5ee6c9d4f117" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:03:57.119315 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e5819c15eac4544a57e5ee6c9d4f117" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:03:57.119798 24546 raft_consensus.cc:3037] T 8e5819c15eac4544a57e5ee6c9d4f117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:57.120239 24471 raft_consensus.cc:3037] T 8e5819c15eac4544a57e5ee6c9d4f117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:57.124442 24921 raft_consensus.cc:504] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:03:57.125946 24514 leader_election.cc:341] T 6c81a1b3b616492ab6c964f45af36077 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:57.126369 24514 leader_election.cc:302] T 6c81a1b3b616492ab6c964f45af36077 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:03:57.130558 24989 raft_consensus.cc:2726] T 6c81a1b3b616492ab6c964f45af36077 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 03:03:57.133450 24546 raft_consensus.cc:2454] T 8e5819c15eac4544a57e5ee6c9d4f117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 1.
I20260925 03:03:57.137647 24586 leader_election.cc:302] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:03:57.138410 24670 raft_consensus.cc:2781] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:03:57.138563 24921 leader_election.cc:288] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:57.142669 24670 raft_consensus.cc:686] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c [term 1 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:03:57.143172 24471 raft_consensus.cc:2454] T 8e5819c15eac4544a57e5ee6c9d4f117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 1.
I20260925 03:03:57.116987 24928 leader_election.cc:288] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 1 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:03:57.143564 24670 consensus_queue.cc:237] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:57.147374 24662 tablet_bootstrap.cc:492] T 25102ba08e4e49388f6176badcd10a25 P eeab93abfdb141a895002ec9f312296c: No bootstrap required, opened a new log
I20260925 03:03:57.147166 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5f9b503b0ab43999d582015a367441e" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:03:57.147811 24662 ts_tablet_manager.cc:1397] T 25102ba08e4e49388f6176badcd10a25 P eeab93abfdb141a895002ec9f312296c: Time spent bootstrapping tablet: real 0.134s user 0.017s sys 0.030s
W20260925 03:03:57.148653 24514 leader_election.cc:341] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:57.149158 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5f9b503b0ab43999d582015a367441e" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:03:57.149857 24620 raft_consensus.cc:3037] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:57.151853 24662 raft_consensus.cc:348] T 25102ba08e4e49388f6176badcd10a25 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:57.152495 24662 raft_consensus.cc:374] T 25102ba08e4e49388f6176badcd10a25 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:57.152776 24662 raft_consensus.cc:729] T 25102ba08e4e49388f6176badcd10a25 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Initialized, Role: FOLLOWER
I20260925 03:03:57.153425 24662 consensus_queue.cc:260] T 25102ba08e4e49388f6176badcd10a25 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:57.156888 24663 tablet_bootstrap.cc:492] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5: No bootstrap required, opened a new log
I20260925 03:03:57.157426 24663 ts_tablet_manager.cc:1397] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent bootstrapping tablet: real 0.113s user 0.011s sys 0.004s
I20260925 03:03:57.153352 24367 catalog_manager.cc:5495] T 11bb46bce3564841b7674e6a693bd2e6 P 869b0a8575b54d08ae2817fad5dd42d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1). New cstate: current_term: 1 leader_uuid: "869b0a8575b54d08ae2817fad5dd42d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: UNKNOWN } } }
I20260925 03:03:57.159190 24662 ts_tablet_manager.cc:1428] T 25102ba08e4e49388f6176badcd10a25 P eeab93abfdb141a895002ec9f312296c: Time spent starting tablet: real 0.011s user 0.005s sys 0.002s
I20260925 03:03:57.160128 24662 tablet_bootstrap.cc:492] T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c: Bootstrap starting.
I20260925 03:03:57.164666 24663 raft_consensus.cc:348] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:57.165867 24663 raft_consensus.cc:374] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:57.166205 24663 raft_consensus.cc:729] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Initialized, Role: FOLLOWER
I20260925 03:03:57.166846 24663 consensus_queue.cc:260] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:57.167368 24662 tablet_bootstrap.cc:654] T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:57.173907 24663 ts_tablet_manager.cc:1428] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent starting tablet: real 0.016s user 0.006s sys 0.000s
I20260925 03:03:57.180893 24663 tablet_bootstrap.cc:492] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Bootstrap starting.
I20260925 03:03:57.184775 24664 tablet_bootstrap.cc:492] T e6b6ec010caf4689b8c90449764f5cf9 P 869b0a8575b54d08ae2817fad5dd42d5: No bootstrap required, opened a new log
I20260925 03:03:57.185237 24664 ts_tablet_manager.cc:1397] T e6b6ec010caf4689b8c90449764f5cf9 P 869b0a8575b54d08ae2817fad5dd42d5: Time spent bootstrapping tablet: real 0.136s user 0.010s sys 0.025s
I20260925 03:03:57.188558 24664 raft_consensus.cc:348] T e6b6ec010caf4689b8c90449764f5cf9 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:57.189282 24664 raft_consensus.cc:374] T e6b6ec010caf4689b8c90449764f5cf9 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:57.189563 24664 raft_consensus.cc:729] T e6b6ec010caf4689b8c90449764f5cf9 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Initialized, Role: FOLLOWER
I20260925 03:03:57.190164 24664 consensus_queue.cc:260] T e6b6ec010caf4689b8c90449764f5cf9 P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:57.192408 24663 tablet_bootstrap.cc:654] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:57.192442 24664 ts_tablet_manager.cc:1428] T e6b6ec010caf4689b8c90449764f5cf9 P 869b0a8575b54d08ae2817fad5dd42d5: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20260925 03:03:57.193537 24664 tablet_bootstrap.cc:492] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5: Bootstrap starting.
I20260925 03:03:57.194829 24670 raft_consensus.cc:482] T 25102ba08e4e49388f6176badcd10a25 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:57.195299 24670 raft_consensus.cc:504] T 25102ba08e4e49388f6176badcd10a25 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:57.198561 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25102ba08e4e49388f6176badcd10a25" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:03:57.199257 24546 raft_consensus.cc:2454] T 25102ba08e4e49388f6176badcd10a25 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 0.
I20260925 03:03:57.200851 24664 tablet_bootstrap.cc:654] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:57.204365 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25102ba08e4e49388f6176badcd10a25" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:03:57.205076 24471 raft_consensus.cc:2454] T 25102ba08e4e49388f6176badcd10a25 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 0.
I20260925 03:03:57.206152 24588 leader_election.cc:302] T 25102ba08e4e49388f6176badcd10a25 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:03:57.206909 24717 raft_consensus.cc:2781] T 25102ba08e4e49388f6176badcd10a25 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 03:03:57.207257 24717 raft_consensus.cc:482] T 25102ba08e4e49388f6176badcd10a25 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:03:57.207553 24717 raft_consensus.cc:3037] T 25102ba08e4e49388f6176badcd10a25 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:57.212736 24989 raft_consensus.cc:482] T c88cbd39e01f44628a094af516be2833 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:57.212939 24670 leader_election.cc:288] T 25102ba08e4e49388f6176badcd10a25 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:03:57.213251 24989 raft_consensus.cc:504] T c88cbd39e01f44628a094af516be2833 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:57.215260 24989 leader_election.cc:288] T c88cbd39e01f44628a094af516be2833 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:57.217448 24618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c88cbd39e01f44628a094af516be2833" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
W20260925 03:03:57.218786 24511 leader_election.cc:341] T c88cbd39e01f44628a094af516be2833 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:57.180609 24620 raft_consensus.cc:2454] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 1.
I20260925 03:03:57.220988 24511 leader_election.cc:302] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:03:57.222137 24931 raft_consensus.cc:2781] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:03:57.222388 24662 tablet_bootstrap.cc:492] T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c: No bootstrap required, opened a new log
I20260925 03:03:57.222643 24931 raft_consensus.cc:686] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 LEADER]: Becoming Leader. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:03:57.222882 24662 ts_tablet_manager.cc:1397] T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c: Time spent bootstrapping tablet: real 0.063s user 0.012s sys 0.007s
I20260925 03:03:57.225819 24662 raft_consensus.cc:348] T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:57.226444 24662 raft_consensus.cc:374] T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:57.226639 24662 raft_consensus.cc:729] T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Initialized, Role: FOLLOWER
I20260925 03:03:57.227342 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c88cbd39e01f44628a094af516be2833" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:03:57.227174 24662 consensus_queue.cc:260] T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:57.227963 24473 raft_consensus.cc:2454] T c88cbd39e01f44628a094af516be2833 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 0.
I20260925 03:03:57.227098 24717 raft_consensus.cc:504] T 25102ba08e4e49388f6176badcd10a25 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:57.229229 24514 leader_election.cc:302] T c88cbd39e01f44628a094af516be2833 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:03:57.230566 24663 tablet_bootstrap.cc:492] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5: No bootstrap required, opened a new log
I20260925 03:03:57.230751 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25102ba08e4e49388f6176badcd10a25" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:03:57.231002 24663 ts_tablet_manager.cc:1397] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent bootstrapping tablet: real 0.056s user 0.013s sys 0.003s
I20260925 03:03:57.231313 24547 raft_consensus.cc:3037] T 25102ba08e4e49388f6176badcd10a25 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:57.231432 24921 raft_consensus.cc:2781] T c88cbd39e01f44628a094af516be2833 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 03:03:57.231774 24921 raft_consensus.cc:482] T c88cbd39e01f44628a094af516be2833 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:03:57.232089 24921 raft_consensus.cc:3037] T c88cbd39e01f44628a094af516be2833 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:57.233458 24662 ts_tablet_manager.cc:1428] T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c: Time spent starting tablet: real 0.010s user 0.002s sys 0.004s
I20260925 03:03:57.235283 24662 tablet_bootstrap.cc:492] T c88cbd39e01f44628a094af516be2833 P eeab93abfdb141a895002ec9f312296c: Bootstrap starting.
I20260925 03:03:57.235304 24931 consensus_queue.cc:237] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:57.236581 24663 raft_consensus.cc:348] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:57.237265 24663 raft_consensus.cc:374] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:57.237669 24663 raft_consensus.cc:729] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Initialized, Role: FOLLOWER
I20260925 03:03:57.237613 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25102ba08e4e49388f6176badcd10a25" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:03:57.238411 24663 consensus_queue.cc:260] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:57.237270 24547 raft_consensus.cc:2454] T 25102ba08e4e49388f6176badcd10a25 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 1.
I20260925 03:03:57.239910 24586 leader_election.cc:302] T 25102ba08e4e49388f6176badcd10a25 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:03:57.240785 24928 raft_consensus.cc:2781] T 25102ba08e4e49388f6176badcd10a25 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:03:57.240875 24663 ts_tablet_manager.cc:1428] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent starting tablet: real 0.010s user 0.007s sys 0.001s
I20260925 03:03:57.241253 24662 tablet_bootstrap.cc:654] T c88cbd39e01f44628a094af516be2833 P eeab93abfdb141a895002ec9f312296c: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:57.241441 24928 raft_consensus.cc:686] T 25102ba08e4e49388f6176badcd10a25 P eeab93abfdb141a895002ec9f312296c [term 1 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:03:57.242039 24663 tablet_bootstrap.cc:492] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Bootstrap starting.
I20260925 03:03:57.242525 24928 consensus_queue.cc:237] T 25102ba08e4e49388f6176badcd10a25 P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:57.246660 24368 catalog_manager.cc:5495] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c reported cstate change: term changed from 0 to 1, leader changed from <none> to eeab93abfdb141a895002ec9f312296c (127.22.193.3). New cstate: current_term: 1 leader_uuid: "eeab93abfdb141a895002ec9f312296c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: HEALTHY } } }
I20260925 03:03:57.239025 24471 raft_consensus.cc:3037] T 25102ba08e4e49388f6176badcd10a25 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:57.238539 24717 leader_election.cc:288] T 25102ba08e4e49388f6176badcd10a25 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 1 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:03:57.257076 24921 raft_consensus.cc:504] T c88cbd39e01f44628a094af516be2833 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:57.263603 24663 tablet_bootstrap.cc:654] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:57.265635 24989 raft_consensus.cc:482] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:57.270282 24368 catalog_manager.cc:5495] T ac9357d144aa49789faaf31fe5e6bf89 P 869b0a8575b54d08ae2817fad5dd42d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1). New cstate: current_term: 1 leader_uuid: "869b0a8575b54d08ae2817fad5dd42d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: UNKNOWN } } }
I20260925 03:03:57.266337 24989 raft_consensus.cc:504] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:57.273183 24662 tablet_bootstrap.cc:492] T c88cbd39e01f44628a094af516be2833 P eeab93abfdb141a895002ec9f312296c: No bootstrap required, opened a new log
I20260925 03:03:57.273725 24662 ts_tablet_manager.cc:1397] T c88cbd39e01f44628a094af516be2833 P eeab93abfdb141a895002ec9f312296c: Time spent bootstrapping tablet: real 0.039s user 0.018s sys 0.009s
I20260925 03:03:57.273478 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c88cbd39e01f44628a094af516be2833" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:03:57.276137 24989 leader_election.cc:288] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:57.277815 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9c33fd100ff44c1a3247319a324a044" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:03:57.278460 24472 raft_consensus.cc:2454] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 0.
I20260925 03:03:57.279104 24662 raft_consensus.cc:348] T c88cbd39e01f44628a094af516be2833 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:57.279860 24662 raft_consensus.cc:374] T c88cbd39e01f44628a094af516be2833 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:57.280097 24662 raft_consensus.cc:729] T c88cbd39e01f44628a094af516be2833 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Initialized, Role: FOLLOWER
I20260925 03:03:57.280628 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9c33fd100ff44c1a3247319a324a044" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:03:57.280747 24662 consensus_queue.cc:260] T c88cbd39e01f44628a094af516be2833 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:57.282538 24514 leader_election.cc:302] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:03:57.281402 24620 raft_consensus.cc:2454] T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 0.
I20260925 03:03:57.290307 24662 ts_tablet_manager.cc:1428] T c88cbd39e01f44628a094af516be2833 P eeab93abfdb141a895002ec9f312296c: Time spent starting tablet: real 0.016s user 0.007s sys 0.000s
I20260925 03:03:57.291236 24662 tablet_bootstrap.cc:492] T 11bb46bce3564841b7674e6a693bd2e6 P eeab93abfdb141a895002ec9f312296c: Bootstrap starting.
I20260925 03:03:57.282823 24473 raft_consensus.cc:3037] T c88cbd39e01f44628a094af516be2833 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:57.290094 24921 leader_election.cc:288] T c88cbd39e01f44628a094af516be2833 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:57.297480 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c88cbd39e01f44628a094af516be2833" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:03:57.298079 24620 raft_consensus.cc:3037] T c88cbd39e01f44628a094af516be2833 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:57.303309 24367 catalog_manager.cc:5495] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2). New cstate: current_term: 1 leader_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: UNKNOWN } } }
I20260925 03:03:57.287693 24989 raft_consensus.cc:2781] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 03:03:57.304888 24989 raft_consensus.cc:482] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:03:57.305224 24989 raft_consensus.cc:3037] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:57.315523 24664 tablet_bootstrap.cc:492] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5: No bootstrap required, opened a new log
I20260925 03:03:57.316097 24664 ts_tablet_manager.cc:1397] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5: Time spent bootstrapping tablet: real 0.123s user 0.006s sys 0.005s
I20260925 03:03:57.304603 24620 raft_consensus.cc:2454] T c88cbd39e01f44628a094af516be2833 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 1.
I20260925 03:03:57.325083 24471 raft_consensus.cc:2454] T 25102ba08e4e49388f6176badcd10a25 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 1.
I20260925 03:03:57.330917 24473 raft_consensus.cc:2454] T c88cbd39e01f44628a094af516be2833 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 1.
I20260925 03:03:57.383354 24514 leader_election.cc:302] T c88cbd39e01f44628a094af516be2833 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:03:57.383029 24664 raft_consensus.cc:348] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:57.385108 24664 raft_consensus.cc:374] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:57.385588 24664 raft_consensus.cc:729] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Initialized, Role: FOLLOWER
I20260925 03:03:57.389966 24814 raft_consensus.cc:2781] T c88cbd39e01f44628a094af516be2833 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:03:57.391427 24663 tablet_bootstrap.cc:492] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5: No bootstrap required, opened a new log
I20260925 03:03:57.391912 24663 ts_tablet_manager.cc:1397] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent bootstrapping tablet: real 0.150s user 0.004s sys 0.019s
I20260925 03:03:57.394644 24664 consensus_queue.cc:260] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:57.397567 24814 raft_consensus.cc:686] T c88cbd39e01f44628a094af516be2833 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 LEADER]: Becoming Leader. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:03:57.397956 24663 raft_consensus.cc:348] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:57.398674 24663 raft_consensus.cc:374] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:57.398365 24814 consensus_queue.cc:237] T c88cbd39e01f44628a094af516be2833 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:57.398957 24663 raft_consensus.cc:729] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Initialized, Role: FOLLOWER
I20260925 03:03:57.399576 24663 consensus_queue.cc:260] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:57.404371 24368 catalog_manager.cc:5495] T 25102ba08e4e49388f6176badcd10a25 P eeab93abfdb141a895002ec9f312296c reported cstate change: term changed from 0 to 1, leader changed from <none> to eeab93abfdb141a895002ec9f312296c (127.22.193.3). New cstate: current_term: 1 leader_uuid: "eeab93abfdb141a895002ec9f312296c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: HEALTHY } } }
I20260925 03:03:57.416725 24663 ts_tablet_manager.cc:1428] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent starting tablet: real 0.024s user 0.006s sys 0.002s
I20260925 03:03:57.417652 24663 tablet_bootstrap.cc:492] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5: Bootstrap starting.
I20260925 03:03:57.420841 24986 raft_consensus.cc:482] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:57.421507 24986 raft_consensus.cc:504] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:57.421731 25003 consensus_queue.cc:1035] T 0be4e0aa984f4f2b81a58e382e434ad5 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 03:03:57.424237 24986 leader_election.cc:288] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:57.424599 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7691c6b475428c82d6181d693d26f3" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:03:57.342633 24989 raft_consensus.cc:504] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:57.427567 24473 raft_consensus.cc:2454] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 0.
I20260925 03:03:57.428987 24514 leader_election.cc:302] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:03:57.432135 24989 leader_election.cc:288] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:57.432557 24664 ts_tablet_manager.cc:1428] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5: Time spent starting tablet: real 0.116s user 0.002s sys 0.006s
I20260925 03:03:57.433619 24664 tablet_bootstrap.cc:492] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5: Bootstrap starting.
I20260925 03:03:57.434903 24989 raft_consensus.cc:2781] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 03:03:57.435024 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9c33fd100ff44c1a3247319a324a044" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:03:57.435389 24989 raft_consensus.cc:482] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:03:57.435673 24473 raft_consensus.cc:3037] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:57.435760 24989 raft_consensus.cc:3037] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:57.445441 24880 consensus_queue.cc:1035] T 21c43af5e6844fdaacc8c625fa01fd8d P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 03:03:57.451908 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9c33fd100ff44c1a3247319a324a044" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:03:57.452512 24622 raft_consensus.cc:3037] T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:57.451463 24473 raft_consensus.cc:2454] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 1.
I20260925 03:03:57.455273 24367 catalog_manager.cc:5495] T c88cbd39e01f44628a094af516be2833 P a04a37fc4aa7460cbfdbeeb4e451e1f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2). New cstate: current_term: 1 leader_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: UNKNOWN } } }
I20260925 03:03:57.458721 24514 leader_election.cc:302] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:03:57.462409 24814 raft_consensus.cc:2781] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader election won for term 1
W20260925 03:03:57.462311 24436 consensus_peers.cc:487] T 11bb46bce3564841b7674e6a693bd2e6 P 869b0a8575b54d08ae2817fad5dd42d5 -> Peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105): Couldn't send request to peer eeab93abfdb141a895002ec9f312296c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260925 03:03:57.467058 24662 tablet_bootstrap.cc:654] T 11bb46bce3564841b7674e6a693bd2e6 P eeab93abfdb141a895002ec9f312296c: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:57.470796 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7691c6b475428c82d6181d693d26f3" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:03:57.475140 24664 tablet_bootstrap.cc:654] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:57.475188 24814 raft_consensus.cc:686] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 LEADER]: Becoming Leader. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:03:57.476150 24814 consensus_queue.cc:237] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:03:57.479928 24511 leader_election.cc:341] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:57.468205 24622 raft_consensus.cc:2454] T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 1.
I20260925 03:03:57.492784 24880 consensus_queue.cc:1035] T 21c43af5e6844fdaacc8c625fa01fd8d P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260925 03:03:57.493952 24989 raft_consensus.cc:504] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:57.495323 24663 tablet_bootstrap.cc:654] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5: Neither blocks nor log segments found. Creating new log.
W20260925 03:03:57.652856 25016 log.cc:924] Time spent T c88cbd39e01f44628a094af516be2833 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.157s user 0.004s sys 0.002s
I20260925 03:03:57.661716 24664 tablet_bootstrap.cc:492] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5: No bootstrap required, opened a new log
I20260925 03:03:57.662279 24664 ts_tablet_manager.cc:1397] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5: Time spent bootstrapping tablet: real 0.229s user 0.022s sys 0.011s
I20260925 03:03:57.663931 24989 leader_election.cc:288] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:57.666175 24664 raft_consensus.cc:348] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:57.666759 24664 raft_consensus.cc:374] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:57.666960 24664 raft_consensus.cc:729] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Initialized, Role: FOLLOWER
I20260925 03:03:57.667606 24664 consensus_queue.cc:260] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:57.672804 24988 raft_consensus.cc:482] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:57.673363 24988 raft_consensus.cc:504] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:57.715909 24664 ts_tablet_manager.cc:1428] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5: Time spent starting tablet: real 0.053s user 0.021s sys 0.007s
I20260925 03:03:57.717178 24662 tablet_bootstrap.cc:492] T 11bb46bce3564841b7674e6a693bd2e6 P eeab93abfdb141a895002ec9f312296c: No bootstrap required, opened a new log
I20260925 03:03:57.717720 24662 ts_tablet_manager.cc:1397] T 11bb46bce3564841b7674e6a693bd2e6 P eeab93abfdb141a895002ec9f312296c: Time spent bootstrapping tablet: real 0.427s user 0.015s sys 0.027s
I20260925 03:03:57.718459 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7691c6b475428c82d6181d693d26f3" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:03:57.719914 24988 leader_election.cc:288] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:57.726933 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6b6ec010caf4689b8c90449764f5cf9" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:03:57.761806 25024 raft_consensus.cc:482] T e6b6ec010caf4689b8c90449764f5cf9 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:57.762445 25024 raft_consensus.cc:504] T e6b6ec010caf4689b8c90449764f5cf9 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:57.741690 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6b6ec010caf4689b8c90449764f5cf9" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:03:57.742599 24471 raft_consensus.cc:3037] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:57.801659 24664 tablet_bootstrap.cc:492] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5: Bootstrap starting.
I20260925 03:03:57.764597 25024 leader_election.cc:288] T e6b6ec010caf4689b8c90449764f5cf9 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
W20260925 03:03:57.810220 24511 leader_election.cc:341] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:57.817642 24663 tablet_bootstrap.cc:492] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5: No bootstrap required, opened a new log
I20260925 03:03:57.818145 24663 ts_tablet_manager.cc:1397] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent bootstrapping tablet: real 0.401s user 0.011s sys 0.035s
I20260925 03:03:57.803032 24472 raft_consensus.cc:2454] T e6b6ec010caf4689b8c90449764f5cf9 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 0.
I20260925 03:03:57.815646 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7691c6b475428c82d6181d693d26f3" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
W20260925 03:03:57.852773 24511 leader_election.cc:341] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105): Illegal state: must be running to vote when last-logged opid is not known
W20260925 03:03:57.855412 25020 log.cc:924] Time spent T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.127s user 0.001s sys 0.000s
I20260925 03:03:57.863633 24663 raft_consensus.cc:348] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:57.864492 24663 raft_consensus.cc:374] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:57.864802 24663 raft_consensus.cc:729] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Initialized, Role: FOLLOWER
I20260925 03:03:57.869931 24663 consensus_queue.cc:260] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:57.872253 24662 raft_consensus.cc:348] T 11bb46bce3564841b7674e6a693bd2e6 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:57.873190 24662 raft_consensus.cc:374] T 11bb46bce3564841b7674e6a693bd2e6 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:57.873464 24662 raft_consensus.cc:729] T 11bb46bce3564841b7674e6a693bd2e6 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Initialized, Role: FOLLOWER
I20260925 03:03:57.874157 24662 consensus_queue.cc:260] T 11bb46bce3564841b7674e6a693bd2e6 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:57.884516 24662 ts_tablet_manager.cc:1428] T 11bb46bce3564841b7674e6a693bd2e6 P eeab93abfdb141a895002ec9f312296c: Time spent starting tablet: real 0.167s user 0.012s sys 0.000s
I20260925 03:03:57.886250 24663 ts_tablet_manager.cc:1428] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent starting tablet: real 0.068s user 0.011s sys 0.005s
I20260925 03:03:57.887377 25032 consensus_queue.cc:1035] T 11bb46bce3564841b7674e6a693bd2e6 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.088s
I20260925 03:03:57.888769 24928 raft_consensus.cc:482] T 11bb46bce3564841b7674e6a693bd2e6 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:57.909988 24614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6b6ec010caf4689b8c90449764f5cf9" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:03:57.889232 24928 raft_consensus.cc:504] T 11bb46bce3564841b7674e6a693bd2e6 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:57.950008 24664 tablet_bootstrap.cc:654] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5: Neither blocks nor log segments found. Creating new log.
W20260925 03:03:57.956300 24436 leader_election.cc:341] T e6b6ec010caf4689b8c90449764f5cf9 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:57.956961 25035 consensus_queue.cc:1035] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.047s
I20260925 03:03:57.997277 24514 leader_election.cc:302] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
W20260925 03:03:58.012071 24436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 54323.7us
W20260925 03:03:58.013387 24436 consensus_peers.cc:487] T ac9357d144aa49789faaf31fe5e6bf89 P 869b0a8575b54d08ae2817fad5dd42d5 -> Peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105): Couldn't send request to peer eeab93abfdb141a895002ec9f312296c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260925 03:03:58.023221 24662 tablet_bootstrap.cc:492] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c: Bootstrap starting.
I20260925 03:03:58.014572 24988 raft_consensus.cc:2781] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 03:03:58.034858 25003 consensus_queue.cc:1035] T 0be4e0aa984f4f2b81a58e382e434ad5 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.018s
I20260925 03:03:58.065855 25042 consensus_queue.cc:1035] T c88cbd39e01f44628a094af516be2833 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.007s
I20260925 03:03:58.045037 24988 raft_consensus.cc:482] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260925 03:03:58.059260 24997 log.cc:924] Time spent T 0be4e0aa984f4f2b81a58e382e434ad5 P 869b0a8575b54d08ae2817fad5dd42d5: Append to log took a long time: real 0.107s user 0.003s sys 0.000s
I20260925 03:03:58.093766 24988 raft_consensus.cc:3037] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:58.110405 25038 consensus_queue.cc:1035] T 25102ba08e4e49388f6176badcd10a25 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.008s
W20260925 03:03:58.113356 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 100249us
I20260925 03:03:58.098083 24928 leader_election.cc:288] T 11bb46bce3564841b7674e6a693bd2e6 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:03:58.097900 24468 raft_consensus.cc:3037] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:58.143296 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bb46bce3564841b7674e6a693bd2e6" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:03:58.144980 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6b6ec010caf4689b8c90449764f5cf9" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:03:58.146104 24471 raft_consensus.cc:2454] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 1.
W20260925 03:03:58.207239 24418 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260925 03:03:58.208796 24664 tablet_bootstrap.cc:492] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5: No bootstrap required, opened a new log
I20260925 03:03:58.209498 24664 ts_tablet_manager.cc:1397] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5: Time spent bootstrapping tablet: real 0.408s user 0.021s sys 0.022s
I20260925 03:03:58.195791 24371 catalog_manager.cc:5495] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2). New cstate: current_term: 1 leader_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: UNKNOWN } } }
I20260925 03:03:58.214162 24514 leader_election.cc:302] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:03:58.234736 24662 tablet_bootstrap.cc:654] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:58.232390 24986 consensus_queue.cc:1035] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20260925 03:03:58.209409 24928 consensus_queue.cc:1035] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.021s
I20260925 03:03:58.240964 25045 raft_consensus.cc:2781] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:03:58.245985 25045 raft_consensus.cc:686] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 LEADER]: Becoming Leader. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:03:58.246853 25045 consensus_queue.cc:237] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:58.261407 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bb46bce3564841b7674e6a693bd2e6" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:03:58.263219 24663 tablet_bootstrap.cc:492] T 420163a6cdce412ebc21c25fe3835638 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Bootstrap starting.
I20260925 03:03:58.270582 25036 consensus_queue.cc:1035] T ac9357d144aa49789faaf31fe5e6bf89 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260925 03:03:58.282202 24664 raft_consensus.cc:348] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:58.283046 24664 raft_consensus.cc:374] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:58.283360 24664 raft_consensus.cc:729] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Initialized, Role: FOLLOWER
I20260925 03:03:58.284026 24664 consensus_queue.cc:260] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:58.288612 24664 ts_tablet_manager.cc:1428] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5: Time spent starting tablet: real 0.079s user 0.005s sys 0.002s
I20260925 03:03:58.294718 25050 consensus_queue.cc:1035] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260925 03:03:58.302060 25026 raft_consensus.cc:482] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:58.303704 24664 tablet_bootstrap.cc:492] T 7b8e0d260a884ecdb20f79de46c36b19 P 869b0a8575b54d08ae2817fad5dd42d5: Bootstrap starting.
W20260925 03:03:58.309224 25056 log.cc:924] Time spent T 11bb46bce3564841b7674e6a693bd2e6 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.083s user 0.001s sys 0.000s
I20260925 03:03:58.313405 24586 leader_election.cc:302] T 11bb46bce3564841b7674e6a693bd2e6 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
W20260925 03:03:58.312417 25057 log.cc:924] Time spent T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.053s user 0.001s sys 0.000s
I20260925 03:03:58.312547 24620 raft_consensus.cc:3037] T 11bb46bce3564841b7674e6a693bd2e6 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:58.312765 25038 consensus_queue.cc:1035] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.016s
I20260925 03:03:58.333760 24988 raft_consensus.cc:504] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:58.335242 24546 raft_consensus.cc:2379] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 in current term 1: Already voted for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in this term.
I20260925 03:03:58.304236 25026 raft_consensus.cc:504] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:58.365037 24663 tablet_bootstrap.cc:654] T 420163a6cdce412ebc21c25fe3835638 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:58.378222 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6b6ec010caf4689b8c90449764f5cf9" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:03:58.378939 24472 raft_consensus.cc:3037] T e6b6ec010caf4689b8c90449764f5cf9 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Advancing to term 1
W20260925 03:03:58.371825 25061 log.cc:924] Time spent T 25102ba08e4e49388f6176badcd10a25 P 869b0a8575b54d08ae2817fad5dd42d5: Append to log took a long time: real 0.079s user 0.001s sys 0.000s
I20260925 03:03:58.397850 24437 leader_election.cc:302] T e6b6ec010caf4689b8c90449764f5cf9 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:03:58.402707 25050 raft_consensus.cc:482] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:58.403231 25038 consensus_queue.cc:1035] T 25102ba08e4e49388f6176badcd10a25 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 03:03:58.411208 25026 leader_election.cc:288] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:58.371385 24988 leader_election.cc:288] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:58.420156 25050 raft_consensus.cc:504] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:58.427958 24664 tablet_bootstrap.cc:654] T 7b8e0d260a884ecdb20f79de46c36b19 P 869b0a8575b54d08ae2817fad5dd42d5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:58.429601 24814 consensus_queue.cc:1035] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.011s
W20260925 03:03:58.446574 25064 log.cc:924] Time spent T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.136s user 0.001s sys 0.000s
I20260925 03:03:58.442009 24928 raft_consensus.cc:2726] T 11bb46bce3564841b7674e6a693bd2e6 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 03:03:58.452330 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5f9b503b0ab43999d582015a367441e" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:03:58.468182 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35992316cb8d4c028333388eff02ecde" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:03:58.468919 24468 raft_consensus.cc:2454] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 0.
I20260925 03:03:58.469498 24618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6b6ec010caf4689b8c90449764f5cf9" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:03:58.470037 24662 tablet_bootstrap.cc:492] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c: No bootstrap required, opened a new log
I20260925 03:03:58.470484 24662 ts_tablet_manager.cc:1397] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c: Time spent bootstrapping tablet: real 0.447s user 0.029s sys 0.026s
I20260925 03:03:58.471920 24514 leader_election.cc:302] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:03:58.474536 25045 raft_consensus.cc:2781] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 03:03:58.474942 25045 raft_consensus.cc:482] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:03:58.475267 24662 raft_consensus.cc:348] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:58.476085 24662 raft_consensus.cc:374] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:58.476349 24662 raft_consensus.cc:729] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Initialized, Role: FOLLOWER
I20260925 03:03:58.476794 24618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35992316cb8d4c028333388eff02ecde" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:03:58.480190 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5f9b503b0ab43999d582015a367441e" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
W20260925 03:03:58.478134 24511 leader_election.cc:341] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:58.475302 25045 raft_consensus.cc:3037] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:58.459995 25050 leader_election.cc:288] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:58.442490 25033 consensus_queue.cc:1035] T c88cbd39e01f44628a094af516be2833 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.025s
I20260925 03:03:58.459932 24472 raft_consensus.cc:2454] T e6b6ec010caf4689b8c90449764f5cf9 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 1.
W20260925 03:03:58.491220 24511 leader_election.cc:341] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:58.479346 24662 consensus_queue.cc:260] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:58.499135 25047 raft_consensus.cc:2726] T e6b6ec010caf4689b8c90449764f5cf9 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 03:03:58.506119 24514 leader_election.cc:302] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:03:58.513093 24921 raft_consensus.cc:2781] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:03:58.607786 24437 leader_election.cc:302] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
W20260925 03:03:58.610284 25069 log.cc:924] Time spent T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5: Append to log took a long time: real 0.114s user 0.000s sys 0.000s
I20260925 03:03:58.616531 24664 tablet_bootstrap.cc:492] T 7b8e0d260a884ecdb20f79de46c36b19 P 869b0a8575b54d08ae2817fad5dd42d5: No bootstrap required, opened a new log
I20260925 03:03:58.617110 24664 ts_tablet_manager.cc:1397] T 7b8e0d260a884ecdb20f79de46c36b19 P 869b0a8575b54d08ae2817fad5dd42d5: Time spent bootstrapping tablet: real 0.314s user 0.016s sys 0.006s
I20260925 03:03:58.615129 24921 raft_consensus.cc:686] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 LEADER]: Becoming Leader. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:03:58.620765 24880 raft_consensus.cc:2726] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 03:03:58.637689 24921 consensus_queue.cc:237] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:58.641669 24663 tablet_bootstrap.cc:492] T 420163a6cdce412ebc21c25fe3835638 P a04a37fc4aa7460cbfdbeeb4e451e1f5: No bootstrap required, opened a new log
I20260925 03:03:58.643989 24663 ts_tablet_manager.cc:1397] T 420163a6cdce412ebc21c25fe3835638 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent bootstrapping tablet: real 0.381s user 0.018s sys 0.019s
I20260925 03:03:58.659245 24663 raft_consensus.cc:348] T 420163a6cdce412ebc21c25fe3835638 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:58.660161 24663 raft_consensus.cc:374] T 420163a6cdce412ebc21c25fe3835638 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:58.660434 24663 raft_consensus.cc:729] T 420163a6cdce412ebc21c25fe3835638 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Initialized, Role: FOLLOWER
I20260925 03:03:58.662273 24664 raft_consensus.cc:348] T 7b8e0d260a884ecdb20f79de46c36b19 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:58.663012 24664 raft_consensus.cc:374] T 7b8e0d260a884ecdb20f79de46c36b19 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:58.663262 24664 raft_consensus.cc:729] T 7b8e0d260a884ecdb20f79de46c36b19 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Initialized, Role: FOLLOWER
I20260925 03:03:58.665688 24664 consensus_queue.cc:260] T 7b8e0d260a884ecdb20f79de46c36b19 P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:58.663911 24663 consensus_queue.cc:260] T 420163a6cdce412ebc21c25fe3835638 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:58.675743 24664 ts_tablet_manager.cc:1428] T 7b8e0d260a884ecdb20f79de46c36b19 P 869b0a8575b54d08ae2817fad5dd42d5: Time spent starting tablet: real 0.058s user 0.007s sys 0.001s
I20260925 03:03:58.695191 25045 raft_consensus.cc:504] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:58.695957 25035 raft_consensus.cc:482] T 6c81a1b3b616492ab6c964f45af36077 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:58.696537 25035 raft_consensus.cc:504] T 6c81a1b3b616492ab6c964f45af36077 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:03:58.740425 24511 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 94780.2us
I20260925 03:03:58.759199 24663 ts_tablet_manager.cc:1428] T 420163a6cdce412ebc21c25fe3835638 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent starting tablet: real 0.115s user 0.012s sys 0.008s
W20260925 03:03:58.772855 25076 log.cc:924] Time spent T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5: Append to log took a long time: real 0.092s user 0.000s sys 0.001s
W20260925 03:03:58.792603 25079 log.cc:924] Time spent T 11bb46bce3564841b7674e6a693bd2e6 P 869b0a8575b54d08ae2817fad5dd42d5: Append to log took a long time: real 0.089s user 0.000s sys 0.001s
I20260925 03:03:58.790699 25035 leader_election.cc:288] T 6c81a1b3b616492ab6c964f45af36077 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:58.794368 25045 leader_election.cc:288] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
W20260925 03:03:58.814330 25074 log.cc:924] Time spent T 8e5819c15eac4544a57e5ee6c9d4f117 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.175s user 0.000s sys 0.001s
I20260925 03:03:58.817943 24989 raft_consensus.cc:482] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:58.852559 24989 raft_consensus.cc:504] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:58.795049 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c81a1b3b616492ab6c964f45af36077" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:03:58.947069 24663 tablet_bootstrap.cc:492] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Bootstrap starting.
I20260925 03:03:58.952219 24989 leader_election.cc:288] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:58.996462 25046 raft_consensus.cc:482] T 7b8e0d260a884ecdb20f79de46c36b19 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:58.998327 25046 raft_consensus.cc:504] T 7b8e0d260a884ecdb20f79de46c36b19 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:59.001537 25046 leader_election.cc:288] T 7b8e0d260a884ecdb20f79de46c36b19 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:59.036779 24928 raft_consensus.cc:482] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:59.037892 24928 raft_consensus.cc:504] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:59.049983 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceab6b0fccd54e909a64f66fe81a9a1f" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
W20260925 03:03:59.060654 24662 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.567s user 0.036s sys 0.035s
W20260925 03:03:59.061501 24662 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.568s user 0.039s sys 0.035s
I20260925 03:03:59.062523 25044 raft_consensus.cc:482] T 420163a6cdce412ebc21c25fe3835638 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:59.063012 24662 ts_tablet_manager.cc:1428] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c: Time spent starting tablet: real 0.592s user 0.046s sys 0.035s
W20260925 03:03:59.063472 24662 ts_tablet_manager.cc:1460] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c: Tablet startup took 592ms
I20260925 03:03:59.063411 25044 raft_consensus.cc:504] T 420163a6cdce412ebc21c25fe3835638 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:03:59.081962 25067 thread.cc:636] apply [worker] (thread pool) Time spent creating pthread: real 0.602s user 0.011s sys 0.019s
W20260925 03:03:59.082512 25067 thread.cc:602] apply [worker] (thread pool) Time spent starting thread: real 0.617s user 0.011s sys 0.019s
I20260925 03:03:59.086098 24664 tablet_bootstrap.cc:492] T cc2df40db08649c9a75e90000b878559 P 869b0a8575b54d08ae2817fad5dd42d5: Bootstrap starting.
I20260925 03:03:59.088425 24928 leader_election.cc:288] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:03:59.091795 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35992316cb8d4c028333388eff02ecde" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
W20260925 03:03:59.092936 24436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 171608us
W20260925 03:03:59.096659 24514 leader_election.cc:341] T 6c81a1b3b616492ab6c964f45af36077 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:59.102741 24470 raft_consensus.cc:3037] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:59.107249 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7691c6b475428c82d6181d693d26f3" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:03:59.136852 25044 leader_election.cc:288] T 420163a6cdce412ebc21c25fe3835638 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
W20260925 03:03:59.064020 24662 ts_tablet_manager.cc:1462] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c: Trace:
0925 03:03:50.298649 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:03:51.287273 (+988624us) service_pool.cc:227] Handling call
0925 03:03:51.289102 (+ 1829us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0925 03:03:51.289192 (+ 90us) ts_tablet_manager.cc:676] Creating new metadata...
0925 03:03:51.320303 (+ 31111us) tablet_metadata.cc:641] Metadata flushed
0925 03:03:51.411038 (+ 90735us) tablet_replica.cc:205] Creating consensus instance
0925 03:03:51.411503 (+ 465us) inbound_call.cc:173] Queueing success response
0925 03:03:58.022931 (+6611428us) ts_tablet_manager.cc:1373] Bootstrapping tablet
0925 03:03:58.470730 (+447799us) ts_tablet_manager.cc:1429] Starting tablet replica
0925 03:03:58.473497 (+ 2767us) tablet_replica.cc:259] Starting instrumentation
0925 03:03:58.473985 (+ 488us) tablet_replica.cc:278] Starting consensus
Metrics: {"raft.queue_time_us":120859,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":131,"tablet-open.queue_time_us":6611505,"thread_start_us":568293,"threads_started":1}
I20260925 03:03:59.139658 24662 tablet_bootstrap.cc:492] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c: Bootstrap starting.
I20260925 03:03:59.139926 24470 raft_consensus.cc:2454] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 1.
I20260925 03:03:59.167484 24663 tablet_bootstrap.cc:654] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:59.174798 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7691c6b475428c82d6181d693d26f3" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:03:59.180998 24615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b8e0d260a884ecdb20f79de46c36b19" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:03:59.183568 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b8e0d260a884ecdb20f79de46c36b19" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:03:59.174125 24464 raft_consensus.cc:2379] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c in current term 1: Already voted for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in this term.
W20260925 03:03:59.173750 24514 leader_election.cc:341] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:59.191045 25089 raft_consensus.cc:482] T cc2df40db08649c9a75e90000b878559 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:59.191836 25089 raft_consensus.cc:504] T cc2df40db08649c9a75e90000b878559 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:59.196553 25089 leader_election.cc:288] T cc2df40db08649c9a75e90000b878559 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:59.203989 24662 tablet_bootstrap.cc:654] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:59.214040 24371 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39374 (request call id 114) took 1397 ms. Trace:
W20260925 03:03:59.216729 24436 leader_election.cc:341] T 7b8e0d260a884ecdb20f79de46c36b19 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:59.217643 24514 leader_election.cc:302] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:03:59.218473 24586 leader_election.cc:302] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:03:59.214548 24371 rpcz_store.cc:276] 0925 03:03:57.827458 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:03:57.827667 (+ 209us) service_pool.cc:227] Handling call
0925 03:03:59.213743 (+1386076us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0925 03:03:58.250011 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0925 03:03:58.250493 (+ 482us) write_op.cc:432] Acquiring schema lock in shared mode
0925 03:03:58.250527 (+ 34us) write_op.cc:435] Acquired schema lock
0925 03:03:58.250542 (+ 15us) tablet.cc:660] Decoding operations
0925 03:03:58.250916 (+ 374us) write_op.cc:620] Acquiring the partition lock for write op
0925 03:03:58.251008 (+ 92us) write_op.cc:641] Partition lock acquired for write op
0925 03:03:58.251036 (+ 28us) tablet.cc:683] Acquiring locks for 1 operations
0925 03:03:58.251216 (+ 180us) tablet.cc:699] Row locks acquired
0925 03:03:58.251233 (+ 17us) write_op.cc:260] PREPARE: finished
0925 03:03:58.251370 (+ 137us) write_op.cc:270] Start()
0925 03:03:58.251480 (+ 110us) write_op.cc:276] Timestamp: P: 1790305438251326 usec, L: 0
0925 03:03:58.251502 (+ 22us) op_driver.cc:331] REPLICATION: starting
0925 03:03:58.289743 (+ 38241us) log.cc:842] Serialized 537 byte log entry
0925 03:03:58.432069 (+142326us) op_driver.cc:444] REPLICATION: finished
0925 03:03:58.855229 (+423160us) spinlock_profiling.cc:243] Waited 5.53 ms on lock 0x7b44000232cc. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f43733ecf11 00007f43734baa95 00007f43734bb4b2 00007f4376163511 00007f437615ee1a 00007f437358a5fe 00007f437c3c1a39 00007f43735a78bb 00007f43735a5cbf 00007f43735a7e22 00007f4379ee0410
0925 03:03:59.133419 (+278190us) write_op.cc:301] APPLY: starting
0925 03:03:59.133574 (+ 155us) tablet.cc:1365] starting BulkCheckPresence
0925 03:03:59.133747 (+ 173us) tablet.cc:1368] finished BulkCheckPresence
0925 03:03:59.133789 (+ 42us) tablet.cc:1370] starting ApplyRowOperation cycle
0925 03:03:59.134238 (+ 449us) tablet.cc:1381] finished ApplyRowOperation cycle
0925 03:03:59.134542 (+ 304us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0925 03:03:59.134596 (+ 54us) write_op.cc:312] APPLY: finished
0925 03:03:59.135064 (+ 468us) log.cc:842] Serialized 22 byte log entry
0925 03:03:59.135300 (+ 236us) write_op.cc:489] Releasing partition, row and schema locks
0925 03:03:59.135457 (+ 157us) write_op.cc:454] Released schema lock
0925 03:03:59.135513 (+ 56us) write_op.cc:341] FINISH: Updating metrics
0925 03:03:59.135902 (+ 389us) write_op.cc:489] Releasing partition, row and schema locks
0925 03:03:59.135941 (+ 39us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":77325,"reported_tablets":3,"child_traces":[["op",{"apply.queue_time_us":668917,"apply.run_cpu_time_us":2988,"apply.run_wall_time_us":20921,"num_ops":1,"prepare.queue_time_us":37461,"prepare.run_cpu_time_us":3128,"prepare.run_wall_time_us":46029,"replication_time_us":175477,"spinlock_wait_cycles":12708224,"thread_start_us":640060,"threads_started":2}]]}
I20260925 03:03:59.229182 24467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "420163a6cdce412ebc21c25fe3835638" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:03:59.237291 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc2df40db08649c9a75e90000b878559" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:03:59.241761 25025 raft_consensus.cc:3037] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Advancing to term 1
W20260925 03:03:59.246778 24514 leader_election.cc:341] T 420163a6cdce412ebc21c25fe3835638 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:59.248632 25042 consensus_queue.cc:1035] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260925 03:03:59.254202 24514 leader_election.cc:341] T cc2df40db08649c9a75e90000b878559 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:59.256120 24664 tablet_bootstrap.cc:654] T cc2df40db08649c9a75e90000b878559 P 869b0a8575b54d08ae2817fad5dd42d5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:59.268929 25034 raft_consensus.cc:2781] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:03:59.269584 25034 raft_consensus.cc:686] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 LEADER]: Becoming Leader. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:03:59.269927 24367 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39382 (request call id 100) took 1239 ms. Trace:
I20260925 03:03:59.270334 24367 rpcz_store.cc:276] 0925 03:03:58.030950 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:03:58.031117 (+ 167us) service_pool.cc:227] Handling call
0925 03:03:59.269748 (+1238631us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":63813,"reported_tablets":1}
I20260925 03:03:59.270612 25034 consensus_queue.cc:237] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:59.301306 24368 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39368 (request call id 106) took 1005 ms. Trace:
I20260925 03:03:59.302471 25025 raft_consensus.cc:2726] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 03:03:59.307729 24618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35992316cb8d4c028333388eff02ecde" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:03:59.309938 24663 tablet_bootstrap.cc:492] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5: No bootstrap required, opened a new log
I20260925 03:03:59.310487 24663 ts_tablet_manager.cc:1397] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent bootstrapping tablet: real 0.364s user 0.016s sys 0.046s
I20260925 03:03:59.313951 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceab6b0fccd54e909a64f66fe81a9a1f" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:03:59.314188 24663 raft_consensus.cc:348] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:59.314955 24663 raft_consensus.cc:374] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:59.315269 24663 raft_consensus.cc:729] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Initialized, Role: FOLLOWER
I20260925 03:03:59.315497 24615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c81a1b3b616492ab6c964f45af36077" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:03:59.315929 24663 consensus_queue.cc:260] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:59.329901 24614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "420163a6cdce412ebc21c25fe3835638" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:03:59.301879 24368 rpcz_store.cc:276] 0925 03:03:58.305534 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:03:58.306787 (+ 1253us) service_pool.cc:227] Handling call
0925 03:03:59.301138 (+994351us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2474,"reported_tablets":5}
I20260925 03:03:59.334424 24614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc2df40db08649c9a75e90000b878559" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:03:59.347352 24663 ts_tablet_manager.cc:1428] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent starting tablet: real 0.037s user 0.008s sys 0.000s
I20260925 03:03:59.348470 24663 tablet_bootstrap.cc:492] T 100c113a9fd84cc8adf2ec7b33ea0093 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Bootstrap starting.
I20260925 03:03:59.362258 24662 tablet_bootstrap.cc:492] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c: No bootstrap required, opened a new log
I20260925 03:03:59.362691 24662 ts_tablet_manager.cc:1397] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c: Time spent bootstrapping tablet: real 0.224s user 0.008s sys 0.013s
W20260925 03:03:59.363646 24437 leader_election.cc:341] T 7b8e0d260a884ecdb20f79de46c36b19 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:59.364003 24437 leader_election.cc:302] T 7b8e0d260a884ecdb20f79de46c36b19 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:03:59.373390 25099 raft_consensus.cc:2726] T 7b8e0d260a884ecdb20f79de46c36b19 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 03:03:59.375135 24664 tablet_bootstrap.cc:492] T cc2df40db08649c9a75e90000b878559 P 869b0a8575b54d08ae2817fad5dd42d5: No bootstrap required, opened a new log
I20260925 03:03:59.375619 24664 ts_tablet_manager.cc:1397] T cc2df40db08649c9a75e90000b878559 P 869b0a8575b54d08ae2817fad5dd42d5: Time spent bootstrapping tablet: real 0.290s user 0.025s sys 0.000s
I20260925 03:03:59.384366 24662 raft_consensus.cc:348] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:59.385262 24662 raft_consensus.cc:374] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:59.385540 24662 raft_consensus.cc:729] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Initialized, Role: FOLLOWER
I20260925 03:03:59.394173 24662 consensus_queue.cc:260] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:59.396991 24662 ts_tablet_manager.cc:1428] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c: Time spent starting tablet: real 0.034s user 0.006s sys 0.001s
I20260925 03:03:59.398201 24662 tablet_bootstrap.cc:492] T ac9357d144aa49789faaf31fe5e6bf89 P eeab93abfdb141a895002ec9f312296c: Bootstrap starting.
I20260925 03:03:59.400553 25084 consensus_queue.cc:1035] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260925 03:03:59.409204 25038 raft_consensus.cc:482] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:59.409703 25038 raft_consensus.cc:504] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:59.412619 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceab6b0fccd54e909a64f66fe81a9a1f" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:03:59.413535 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceab6b0fccd54e909a64f66fe81a9a1f" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:03:59.413695 24664 raft_consensus.cc:348] T cc2df40db08649c9a75e90000b878559 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:59.414283 24544 raft_consensus.cc:2454] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 0.
W20260925 03:03:59.414441 24588 leader_election.cc:341] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:59.414444 24664 raft_consensus.cc:374] T cc2df40db08649c9a75e90000b878559 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:59.415259 24664 raft_consensus.cc:729] T cc2df40db08649c9a75e90000b878559 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Initialized, Role: FOLLOWER
I20260925 03:03:59.415845 24664 consensus_queue.cc:260] T cc2df40db08649c9a75e90000b878559 P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:59.412580 25038 leader_election.cc:288] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
W20260925 03:03:59.409086 24511 consensus_peers.cc:487] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 -> Peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105): Couldn't send request to peer eeab93abfdb141a895002ec9f312296c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260925 03:03:59.418283 24586 leader_election.cc:302] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5
W20260925 03:03:59.418663 24511 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 60023.3us
W20260925 03:03:59.419693 24511 leader_election.cc:341] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:59.420030 25038 raft_consensus.cc:2781] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 03:03:59.420203 24664 ts_tablet_manager.cc:1428] T cc2df40db08649c9a75e90000b878559 P 869b0a8575b54d08ae2817fad5dd42d5: Time spent starting tablet: real 0.044s user 0.007s sys 0.001s
I20260925 03:03:59.420476 25038 raft_consensus.cc:482] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:03:59.420841 25038 raft_consensus.cc:3037] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:59.421353 24664 tablet_bootstrap.cc:492] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5: Bootstrap starting.
W20260925 03:03:59.425035 24511 leader_election.cc:341] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:59.425568 24663 tablet_bootstrap.cc:654] T 100c113a9fd84cc8adf2ec7b33ea0093 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:59.425534 24511 leader_election.cc:302] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
W20260925 03:03:59.434504 24511 leader_election.cc:341] T 6c81a1b3b616492ab6c964f45af36077 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:59.434847 24511 leader_election.cc:302] T 6c81a1b3b616492ab6c964f45af36077 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:03:59.433588 25084 raft_consensus.cc:2726] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 03:03:59.438534 25095 raft_consensus.cc:2726] T 6c81a1b3b616492ab6c964f45af36077 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260925 03:03:59.445468 24511 leader_election.cc:341] T 420163a6cdce412ebc21c25fe3835638 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:59.447373 24511 leader_election.cc:302] T 420163a6cdce412ebc21c25fe3835638 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:03:59.449223 25095 raft_consensus.cc:2726] T 420163a6cdce412ebc21c25fe3835638 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260925 03:03:59.450026 24511 leader_election.cc:341] T cc2df40db08649c9a75e90000b878559 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:59.450431 24511 leader_election.cc:302] T cc2df40db08649c9a75e90000b878559 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:03:59.457741 25095 raft_consensus.cc:2726] T cc2df40db08649c9a75e90000b878559 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 03:03:59.464038 24662 tablet_bootstrap.cc:654] T ac9357d144aa49789faaf31fe5e6bf89 P eeab93abfdb141a895002ec9f312296c: Neither blocks nor log segments found. Creating new log.
W20260925 03:03:59.476922 24511 consensus_peers.cc:487] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 -> Peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105): Couldn't send request to peer eeab93abfdb141a895002ec9f312296c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260925 03:03:59.482636 24664 tablet_bootstrap.cc:654] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5: Neither blocks nor log segments found. Creating new log.
W20260925 03:03:59.483261 24418 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260925 03:03:59.484308 24368 catalog_manager.cc:5495] T e6b6ec010caf4689b8c90449764f5cf9 P 869b0a8575b54d08ae2817fad5dd42d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2). New cstate: current_term: 1 leader_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } }
I20260925 03:03:59.486058 25038 raft_consensus.cc:504] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:59.489698 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceab6b0fccd54e909a64f66fe81a9a1f" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
W20260925 03:03:59.491511 24588 leader_election.cc:341] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:59.507155 24662 tablet_bootstrap.cc:492] T ac9357d144aa49789faaf31fe5e6bf89 P eeab93abfdb141a895002ec9f312296c: No bootstrap required, opened a new log
I20260925 03:03:59.507696 24662 ts_tablet_manager.cc:1397] T ac9357d144aa49789faaf31fe5e6bf89 P eeab93abfdb141a895002ec9f312296c: Time spent bootstrapping tablet: real 0.110s user 0.010s sys 0.007s
I20260925 03:03:59.510324 24662 raft_consensus.cc:348] T ac9357d144aa49789faaf31fe5e6bf89 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:59.510725 25038 leader_election.cc:288] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 1 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:03:59.511878 24662 raft_consensus.cc:374] T ac9357d144aa49789faaf31fe5e6bf89 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:59.512202 24662 raft_consensus.cc:729] T ac9357d144aa49789faaf31fe5e6bf89 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Initialized, Role: FOLLOWER
I20260925 03:03:59.514737 24663 tablet_bootstrap.cc:492] T 100c113a9fd84cc8adf2ec7b33ea0093 P a04a37fc4aa7460cbfdbeeb4e451e1f5: No bootstrap required, opened a new log
I20260925 03:03:59.514608 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceab6b0fccd54e909a64f66fe81a9a1f" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:03:59.515236 24663 ts_tablet_manager.cc:1397] T 100c113a9fd84cc8adf2ec7b33ea0093 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent bootstrapping tablet: real 0.167s user 0.011s sys 0.008s
I20260925 03:03:59.518361 24663 raft_consensus.cc:348] T 100c113a9fd84cc8adf2ec7b33ea0093 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:59.519088 24663 raft_consensus.cc:374] T 100c113a9fd84cc8adf2ec7b33ea0093 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:59.519337 24663 raft_consensus.cc:729] T 100c113a9fd84cc8adf2ec7b33ea0093 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Initialized, Role: FOLLOWER
I20260925 03:03:59.520107 24662 consensus_queue.cc:260] T ac9357d144aa49789faaf31fe5e6bf89 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:59.515596 24543 raft_consensus.cc:3037] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:59.528131 24662 ts_tablet_manager.cc:1428] T ac9357d144aa49789faaf31fe5e6bf89 P eeab93abfdb141a895002ec9f312296c: Time spent starting tablet: real 0.020s user 0.008s sys 0.000s
I20260925 03:03:59.529175 24662 tablet_bootstrap.cc:492] T cc2df40db08649c9a75e90000b878559 P eeab93abfdb141a895002ec9f312296c: Bootstrap starting.
I20260925 03:03:59.530719 24663 consensus_queue.cc:260] T 100c113a9fd84cc8adf2ec7b33ea0093 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:59.535049 24367 catalog_manager.cc:5495] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2). New cstate: current_term: 1 leader_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: UNKNOWN } } }
I20260925 03:03:59.537142 24367 catalog_manager.cc:5495] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2). New cstate: current_term: 1 leader_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: UNKNOWN } } }
I20260925 03:03:59.541354 24663 ts_tablet_manager.cc:1428] T 100c113a9fd84cc8adf2ec7b33ea0093 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent starting tablet: real 0.025s user 0.008s sys 0.000s
I20260925 03:03:59.549397 24662 tablet_bootstrap.cc:654] T cc2df40db08649c9a75e90000b878559 P eeab93abfdb141a895002ec9f312296c: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:59.567209 24662 tablet_bootstrap.cc:492] T cc2df40db08649c9a75e90000b878559 P eeab93abfdb141a895002ec9f312296c: No bootstrap required, opened a new log
I20260925 03:03:59.567710 24662 ts_tablet_manager.cc:1397] T cc2df40db08649c9a75e90000b878559 P eeab93abfdb141a895002ec9f312296c: Time spent bootstrapping tablet: real 0.039s user 0.007s sys 0.009s
I20260925 03:03:59.570250 25089 raft_consensus.cc:482] T 100c113a9fd84cc8adf2ec7b33ea0093 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:59.570776 25089 raft_consensus.cc:504] T 100c113a9fd84cc8adf2ec7b33ea0093 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:59.570153 24543 raft_consensus.cc:2454] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 1.
I20260925 03:03:59.574354 24586 leader_election.cc:302] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:03:59.572355 24664 tablet_bootstrap.cc:492] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5: No bootstrap required, opened a new log
I20260925 03:03:59.575181 25083 raft_consensus.cc:2781] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:03:59.575315 24664 ts_tablet_manager.cc:1397] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5: Time spent bootstrapping tablet: real 0.154s user 0.032s sys 0.049s
I20260925 03:03:59.577844 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "100c113a9fd84cc8adf2ec7b33ea0093" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:03:59.578066 24614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "100c113a9fd84cc8adf2ec7b33ea0093" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
W20260925 03:03:59.580032 24514 leader_election.cc:341] T 100c113a9fd84cc8adf2ec7b33ea0093 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187): Illegal state: must be running to vote when last-logged opid is not known
W20260925 03:03:59.580637 24511 leader_election.cc:341] T 100c113a9fd84cc8adf2ec7b33ea0093 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:59.581115 24511 leader_election.cc:302] T 100c113a9fd84cc8adf2ec7b33ea0093 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:03:59.572819 25089 leader_election.cc:288] T 100c113a9fd84cc8adf2ec7b33ea0093 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:59.582484 25089 raft_consensus.cc:2726] T 100c113a9fd84cc8adf2ec7b33ea0093 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 03:03:59.573342 25034 raft_consensus.cc:482] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:59.583539 25034 raft_consensus.cc:504] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:59.584995 25083 raft_consensus.cc:686] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [term 1 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:03:59.585532 25034 leader_election.cc:288] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:59.585914 25083 consensus_queue.cc:237] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:59.587954 24614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b8e0d260a884ecdb20f79de46c36b19" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
W20260925 03:03:59.589432 24511 leader_election.cc:341] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:59.591179 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b8e0d260a884ecdb20f79de46c36b19" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:03:59.591825 24464 raft_consensus.cc:2454] T 7b8e0d260a884ecdb20f79de46c36b19 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 0.
I20260925 03:03:59.594060 24514 leader_election.cc:302] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:03:59.594997 25089 raft_consensus.cc:2781] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 03:03:59.595360 25089 raft_consensus.cc:482] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:03:59.595651 25089 raft_consensus.cc:3037] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:59.605350 24662 raft_consensus.cc:348] T cc2df40db08649c9a75e90000b878559 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:59.607147 24662 raft_consensus.cc:374] T cc2df40db08649c9a75e90000b878559 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:59.607434 24662 raft_consensus.cc:729] T cc2df40db08649c9a75e90000b878559 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Initialized, Role: FOLLOWER
I20260925 03:03:59.608101 24662 consensus_queue.cc:260] T cc2df40db08649c9a75e90000b878559 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:59.604197 24664 raft_consensus.cc:348] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:59.610205 24664 raft_consensus.cc:374] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:59.610535 24664 raft_consensus.cc:729] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Initialized, Role: FOLLOWER
I20260925 03:03:59.612454 24664 consensus_queue.cc:260] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:59.614113 24664 ts_tablet_manager.cc:1428] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5: Time spent starting tablet: real 0.038s user 0.005s sys 0.000s
I20260925 03:03:59.615072 24664 tablet_bootstrap.cc:492] T ceab6b0fccd54e909a64f66fe81a9a1f P 869b0a8575b54d08ae2817fad5dd42d5: Bootstrap starting.
I20260925 03:03:59.620146 24662 ts_tablet_manager.cc:1428] T cc2df40db08649c9a75e90000b878559 P eeab93abfdb141a895002ec9f312296c: Time spent starting tablet: real 0.052s user 0.006s sys 0.000s
I20260925 03:03:59.621937 24662 tablet_bootstrap.cc:492] T e6b6ec010caf4689b8c90449764f5cf9 P eeab93abfdb141a895002ec9f312296c: Bootstrap starting.
I20260925 03:03:59.636797 24664 tablet_bootstrap.cc:654] T ceab6b0fccd54e909a64f66fe81a9a1f P 869b0a8575b54d08ae2817fad5dd42d5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:59.639200 24662 tablet_bootstrap.cc:654] T e6b6ec010caf4689b8c90449764f5cf9 P eeab93abfdb141a895002ec9f312296c: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:59.647332 25089 raft_consensus.cc:504] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:59.650233 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b8e0d260a884ecdb20f79de46c36b19" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:03:59.650872 24464 raft_consensus.cc:3037] T 7b8e0d260a884ecdb20f79de46c36b19 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:59.662633 24614 raft_consensus.cc:3037] T ac9357d144aa49789faaf31fe5e6bf89 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:59.678714 24664 tablet_bootstrap.cc:492] T ceab6b0fccd54e909a64f66fe81a9a1f P 869b0a8575b54d08ae2817fad5dd42d5: No bootstrap required, opened a new log
I20260925 03:03:59.679348 24664 ts_tablet_manager.cc:1397] T ceab6b0fccd54e909a64f66fe81a9a1f P 869b0a8575b54d08ae2817fad5dd42d5: Time spent bootstrapping tablet: real 0.064s user 0.012s sys 0.003s
I20260925 03:03:59.698067 24464 raft_consensus.cc:2454] T 7b8e0d260a884ecdb20f79de46c36b19 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 1.
I20260925 03:03:59.699208 24514 leader_election.cc:302] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:03:59.700120 25044 raft_consensus.cc:2781] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:03:59.700201 25026 raft_consensus.cc:482] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:59.700632 25044 raft_consensus.cc:686] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 LEADER]: Becoming Leader. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:03:59.701483 25044 consensus_queue.cc:237] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:59.700802 25026 raft_consensus.cc:504] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:59.705189 25026 leader_election.cc:288] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:59.705484 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "420163a6cdce412ebc21c25fe3835638" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
W20260925 03:03:59.706998 24436 leader_election.cc:341] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:59.708320 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "420163a6cdce412ebc21c25fe3835638" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:03:59.709105 24544 raft_consensus.cc:2454] T 420163a6cdce412ebc21c25fe3835638 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 0.
I20260925 03:03:59.712317 24437 leader_election.cc:302] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:03:59.713451 25099 raft_consensus.cc:2781] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 03:03:59.713831 25099 raft_consensus.cc:482] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:03:59.714165 25099 raft_consensus.cc:3037] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:59.722571 25099 raft_consensus.cc:504] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:59.725533 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "420163a6cdce412ebc21c25fe3835638" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
W20260925 03:03:59.726720 24436 leader_election.cc:341] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:59.733206 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "420163a6cdce412ebc21c25fe3835638" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:03:59.734009 24544 raft_consensus.cc:3037] T 420163a6cdce412ebc21c25fe3835638 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:59.735325 25089 leader_election.cc:288] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:59.745280 24664 raft_consensus.cc:348] T ceab6b0fccd54e909a64f66fe81a9a1f P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:59.746680 24664 raft_consensus.cc:374] T ceab6b0fccd54e909a64f66fe81a9a1f P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:59.747072 24664 raft_consensus.cc:729] T ceab6b0fccd54e909a64f66fe81a9a1f P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Initialized, Role: FOLLOWER
I20260925 03:03:59.740545 25099 leader_election.cc:288] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:59.747715 24664 consensus_queue.cc:260] T ceab6b0fccd54e909a64f66fe81a9a1f P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:59.764391 24664 ts_tablet_manager.cc:1428] T ceab6b0fccd54e909a64f66fe81a9a1f P 869b0a8575b54d08ae2817fad5dd42d5: Time spent starting tablet: real 0.085s user 0.007s sys 0.001s
I20260925 03:03:59.764161 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b8e0d260a884ecdb20f79de46c36b19" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
W20260925 03:03:59.766330 24511 leader_election.cc:341] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105): Illegal state: must be running to vote when last-logged opid is not known
I20260925 03:03:59.769788 24662 tablet_bootstrap.cc:492] T e6b6ec010caf4689b8c90449764f5cf9 P eeab93abfdb141a895002ec9f312296c: No bootstrap required, opened a new log
I20260925 03:03:59.770252 24662 ts_tablet_manager.cc:1397] T e6b6ec010caf4689b8c90449764f5cf9 P eeab93abfdb141a895002ec9f312296c: Time spent bootstrapping tablet: real 0.149s user 0.026s sys 0.042s
I20260925 03:03:59.772172 24662 raft_consensus.cc:348] T e6b6ec010caf4689b8c90449764f5cf9 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:59.776270 24662 raft_consensus.cc:374] T e6b6ec010caf4689b8c90449764f5cf9 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:59.776572 24662 raft_consensus.cc:729] T e6b6ec010caf4689b8c90449764f5cf9 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Initialized, Role: FOLLOWER
I20260925 03:03:59.780216 24664 tablet_bootstrap.cc:492] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5: Bootstrap starting.
I20260925 03:03:59.780023 24367 catalog_manager.cc:5495] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c reported cstate change: term changed from 0 to 1, leader changed from <none> to eeab93abfdb141a895002ec9f312296c (127.22.193.3). New cstate: current_term: 1 leader_uuid: "eeab93abfdb141a895002ec9f312296c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: HEALTHY } } }
I20260925 03:03:59.782529 24662 consensus_queue.cc:260] T e6b6ec010caf4689b8c90449764f5cf9 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:59.788295 24662 ts_tablet_manager.cc:1428] T e6b6ec010caf4689b8c90449764f5cf9 P eeab93abfdb141a895002ec9f312296c: Time spent starting tablet: real 0.018s user 0.005s sys 0.002s
I20260925 03:03:59.789106 24662 tablet_bootstrap.cc:492] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c: Bootstrap starting.
I20260925 03:03:59.798664 24544 raft_consensus.cc:2454] T 420163a6cdce412ebc21c25fe3835638 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 1.
I20260925 03:03:59.800927 24437 leader_election.cc:302] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:03:59.801729 25026 raft_consensus.cc:2781] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:03:59.804620 25026 raft_consensus.cc:686] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 LEADER]: Becoming Leader. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:03:59.807215 25026 consensus_queue.cc:237] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:59.808285 24662 tablet_bootstrap.cc:654] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:59.814927 25030 raft_consensus.cc:482] T cc2df40db08649c9a75e90000b878559 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:59.815429 25030 raft_consensus.cc:504] T cc2df40db08649c9a75e90000b878559 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:59.817899 25030 leader_election.cc:288] T cc2df40db08649c9a75e90000b878559 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:59.820057 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc2df40db08649c9a75e90000b878559" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:03:59.820719 24622 raft_consensus.cc:2454] T cc2df40db08649c9a75e90000b878559 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 0.
I20260925 03:03:59.818602 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc2df40db08649c9a75e90000b878559" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:03:59.822121 24436 leader_election.cc:302] T cc2df40db08649c9a75e90000b878559 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:03:59.822593 24544 raft_consensus.cc:2454] T cc2df40db08649c9a75e90000b878559 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 0.
I20260925 03:03:59.823014 25101 raft_consensus.cc:2781] T cc2df40db08649c9a75e90000b878559 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 03:03:59.823395 25101 raft_consensus.cc:482] T cc2df40db08649c9a75e90000b878559 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:03:59.823688 25101 raft_consensus.cc:3037] T cc2df40db08649c9a75e90000b878559 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:59.825104 24369 catalog_manager.cc:5495] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1). New cstate: current_term: 1 leader_uuid: "869b0a8575b54d08ae2817fad5dd42d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: UNKNOWN } } }
I20260925 03:03:59.848198 24928 raft_consensus.cc:482] T e6b6ec010caf4689b8c90449764f5cf9 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:59.848768 24928 raft_consensus.cc:504] T e6b6ec010caf4689b8c90449764f5cf9 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:59.849570 24367 catalog_manager.cc:5495] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2). New cstate: current_term: 1 leader_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: UNKNOWN } } }
I20260925 03:03:59.850710 24928 leader_election.cc:288] T e6b6ec010caf4689b8c90449764f5cf9 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:03:59.852586 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6b6ec010caf4689b8c90449764f5cf9" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:03:59.853248 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6b6ec010caf4689b8c90449764f5cf9" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:03:59.854871 24588 leader_election.cc:302] T e6b6ec010caf4689b8c90449764f5cf9 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:03:59.856426 24928 raft_consensus.cc:2726] T e6b6ec010caf4689b8c90449764f5cf9 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260925 03:03:59.869508 24511 consensus_peers.cc:487] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 -> Peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105): Couldn't send request to peer eeab93abfdb141a895002ec9f312296c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260925 03:03:59.875891 24664 tablet_bootstrap.cc:654] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:59.876466 24615 raft_consensus.cc:3037] T e6b6ec010caf4689b8c90449764f5cf9 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:59.885632 25101 raft_consensus.cc:504] T cc2df40db08649c9a75e90000b878559 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:59.903174 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc2df40db08649c9a75e90000b878559" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:03:59.904479 24544 raft_consensus.cc:3037] T cc2df40db08649c9a75e90000b878559 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:59.905555 25038 raft_consensus.cc:482] T ac9357d144aa49789faaf31fe5e6bf89 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:03:59.905942 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc2df40db08649c9a75e90000b878559" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:03:59.906597 24617 raft_consensus.cc:3037] T cc2df40db08649c9a75e90000b878559 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:03:59.906191 25038 raft_consensus.cc:504] T ac9357d144aa49789faaf31fe5e6bf89 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:59.903956 25101 leader_election.cc:288] T cc2df40db08649c9a75e90000b878559 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:03:59.913487 24544 raft_consensus.cc:2454] T cc2df40db08649c9a75e90000b878559 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 1.
I20260925 03:03:59.915310 24437 leader_election.cc:302] T cc2df40db08649c9a75e90000b878559 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:03:59.916227 25101 raft_consensus.cc:2781] T cc2df40db08649c9a75e90000b878559 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:03:59.917493 25101 raft_consensus.cc:686] T cc2df40db08649c9a75e90000b878559 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 LEADER]: Becoming Leader. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:03:59.918337 25101 consensus_queue.cc:237] T cc2df40db08649c9a75e90000b878559 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:59.919602 25044 consensus_queue.cc:1035] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260925 03:03:59.924448 25038 leader_election.cc:288] T ac9357d144aa49789faaf31fe5e6bf89 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:03:59.925207 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9357d144aa49789faaf31fe5e6bf89" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:03:59.932962 24617 raft_consensus.cc:2454] T cc2df40db08649c9a75e90000b878559 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 1.
I20260925 03:03:59.947970 24662 tablet_bootstrap.cc:492] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c: No bootstrap required, opened a new log
I20260925 03:03:59.948140 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9357d144aa49789faaf31fe5e6bf89" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:03:59.948462 24662 ts_tablet_manager.cc:1397] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c: Time spent bootstrapping tablet: real 0.160s user 0.031s sys 0.049s
I20260925 03:03:59.950168 24586 leader_election.cc:302] T ac9357d144aa49789faaf31fe5e6bf89 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:03:59.951282 25038 raft_consensus.cc:2726] T ac9357d144aa49789faaf31fe5e6bf89 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 03:03:59.957099 24662 raft_consensus.cc:348] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:59.957847 24662 raft_consensus.cc:374] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:59.958123 24662 raft_consensus.cc:729] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Initialized, Role: FOLLOWER
I20260925 03:03:59.958737 24662 consensus_queue.cc:260] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:59.967701 24664 tablet_bootstrap.cc:492] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5: No bootstrap required, opened a new log
I20260925 03:03:59.968410 24664 ts_tablet_manager.cc:1397] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5: Time spent bootstrapping tablet: real 0.188s user 0.013s sys 0.000s
I20260925 03:03:59.969142 24369 catalog_manager.cc:5495] T cc2df40db08649c9a75e90000b878559 P 869b0a8575b54d08ae2817fad5dd42d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1). New cstate: current_term: 1 leader_uuid: "869b0a8575b54d08ae2817fad5dd42d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: UNKNOWN } } }
I20260925 03:03:59.970865 24664 raft_consensus.cc:348] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:59.971587 24664 raft_consensus.cc:374] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:03:59.971868 24664 raft_consensus.cc:729] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Initialized, Role: FOLLOWER
I20260925 03:03:59.972483 24662 ts_tablet_manager.cc:1428] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c: Time spent starting tablet: real 0.023s user 0.005s sys 0.000s
I20260925 03:03:59.973573 24662 tablet_bootstrap.cc:492] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c: Bootstrap starting.
I20260925 03:03:59.972508 24664 consensus_queue.cc:260] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:59.979770 24662 tablet_bootstrap.cc:654] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:59.980979 24664 ts_tablet_manager.cc:1428] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5: Time spent starting tablet: real 0.012s user 0.006s sys 0.000s
I20260925 03:03:59.982613 24664 tablet_bootstrap.cc:492] T 100c113a9fd84cc8adf2ec7b33ea0093 P 869b0a8575b54d08ae2817fad5dd42d5: Bootstrap starting.
I20260925 03:03:59.986902 25038 raft_consensus.cc:482] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:03:59.989374 25038 raft_consensus.cc:504] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:03:59.991425 25038 leader_election.cc:288] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:03:59.992470 24664 tablet_bootstrap.cc:654] T 100c113a9fd84cc8adf2ec7b33ea0093 P 869b0a8575b54d08ae2817fad5dd42d5: Neither blocks nor log segments found. Creating new log.
I20260925 03:03:59.993417 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c81a1b3b616492ab6c964f45af36077" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:03:59.994120 24544 raft_consensus.cc:2454] T 6c81a1b3b616492ab6c964f45af36077 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 0.
I20260925 03:03:59.996203 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c81a1b3b616492ab6c964f45af36077" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:03:59.996989 24470 raft_consensus.cc:2454] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 0.
I20260925 03:03:59.997769 24662 tablet_bootstrap.cc:492] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c: No bootstrap required, opened a new log
I20260925 03:03:59.998257 24662 ts_tablet_manager.cc:1397] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c: Time spent bootstrapping tablet: real 0.025s user 0.012s sys 0.003s
I20260925 03:03:59.998113 24588 leader_election.cc:302] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:00.001032 24470 raft_consensus.cc:3037] T ceab6b0fccd54e909a64f66fe81a9a1f P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:04:00.001204 24662 raft_consensus.cc:348] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:00.001988 24662 raft_consensus.cc:374] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:04:00.002230 24662 raft_consensus.cc:729] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Initialized, Role: FOLLOWER
I20260925 03:04:00.002846 24662 consensus_queue.cc:260] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:00.016393 25083 raft_consensus.cc:2781] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 03:04:00.016764 24662 ts_tablet_manager.cc:1428] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c: Time spent starting tablet: real 0.018s user 0.005s sys 0.003s
I20260925 03:04:00.016733 25025 raft_consensus.cc:482] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:00.017323 25025 raft_consensus.cc:504] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:00.017906 24662 tablet_bootstrap.cc:492] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c: Bootstrap starting.
I20260925 03:04:00.016861 25083 raft_consensus.cc:482] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:00.019371 25083 raft_consensus.cc:3037] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:04:00.020869 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b8e0d260a884ecdb20f79de46c36b19" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:00.022904 24468 raft_consensus.cc:2379] T 7b8e0d260a884ecdb20f79de46c36b19 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c in current term 1: Already voted for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in this term.
I20260925 03:04:00.023284 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b8e0d260a884ecdb20f79de46c36b19" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:00.019407 25025 leader_election.cc:288] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:00.029253 24586 leader_election.cc:302] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:00.029412 25083 raft_consensus.cc:504] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:00.030220 25025 raft_consensus.cc:3037] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:04:00.033355 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c81a1b3b616492ab6c964f45af36077" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:00.033969 24468 raft_consensus.cc:3037] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:04:00.036233 24662 tablet_bootstrap.cc:654] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c: Neither blocks nor log segments found. Creating new log.
I20260925 03:04:00.041177 25083 leader_election.cc:288] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 1 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:00.049075 25026 raft_consensus.cc:482] T ceab6b0fccd54e909a64f66fe81a9a1f P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:00.050112 25026 raft_consensus.cc:504] T ceab6b0fccd54e909a64f66fe81a9a1f P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:00.050463 24468 raft_consensus.cc:2454] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 1.
I20260925 03:04:00.051573 24588 leader_election.cc:302] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:00.052485 24928 raft_consensus.cc:2781] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:04:00.053531 24664 tablet_bootstrap.cc:492] T 100c113a9fd84cc8adf2ec7b33ea0093 P 869b0a8575b54d08ae2817fad5dd42d5: No bootstrap required, opened a new log
I20260925 03:04:00.048031 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c81a1b3b616492ab6c964f45af36077" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:00.054026 24664 ts_tablet_manager.cc:1397] T 100c113a9fd84cc8adf2ec7b33ea0093 P 869b0a8575b54d08ae2817fad5dd42d5: Time spent bootstrapping tablet: real 0.072s user 0.016s sys 0.042s
I20260925 03:04:00.053905 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceab6b0fccd54e909a64f66fe81a9a1f" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:00.054361 24544 raft_consensus.cc:3037] T 6c81a1b3b616492ab6c964f45af36077 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:04:00.053568 24615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceab6b0fccd54e909a64f66fe81a9a1f" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:00.053575 25026 leader_election.cc:288] T ceab6b0fccd54e909a64f66fe81a9a1f P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:00.056936 24543 raft_consensus.cc:2454] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 1.
I20260925 03:04:00.061558 24437 leader_election.cc:302] T ceab6b0fccd54e909a64f66fe81a9a1f P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:04:00.063309 25101 raft_consensus.cc:2781] T ceab6b0fccd54e909a64f66fe81a9a1f P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 03:04:00.064334 25101 raft_consensus.cc:482] T ceab6b0fccd54e909a64f66fe81a9a1f P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:00.069501 25025 raft_consensus.cc:2726] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 03:04:00.051092 25083 consensus_queue.cc:1035] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.019s
I20260925 03:04:00.064642 25101 raft_consensus.cc:3037] T ceab6b0fccd54e909a64f66fe81a9a1f P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:00.070482 24928 raft_consensus.cc:686] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c [term 1 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:00.078328 24928 consensus_queue.cc:237] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:00.079938 24664 raft_consensus.cc:348] T 100c113a9fd84cc8adf2ec7b33ea0093 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:00.080633 24664 raft_consensus.cc:374] T 100c113a9fd84cc8adf2ec7b33ea0093 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:04:00.081985 24664 raft_consensus.cc:729] T 100c113a9fd84cc8adf2ec7b33ea0093 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Initialized, Role: FOLLOWER
I20260925 03:04:00.082602 24664 consensus_queue.cc:260] T 100c113a9fd84cc8adf2ec7b33ea0093 P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:00.084959 24664 ts_tablet_manager.cc:1428] T 100c113a9fd84cc8adf2ec7b33ea0093 P 869b0a8575b54d08ae2817fad5dd42d5: Time spent starting tablet: real 0.031s user 0.008s sys 0.000s
I20260925 03:04:00.086027 24544 raft_consensus.cc:2454] T 6c81a1b3b616492ab6c964f45af36077 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 1.
I20260925 03:04:00.090111 24662 tablet_bootstrap.cc:492] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c: No bootstrap required, opened a new log
I20260925 03:04:00.090584 24662 ts_tablet_manager.cc:1397] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c: Time spent bootstrapping tablet: real 0.073s user 0.008s sys 0.011s
I20260925 03:04:00.091427 25101 raft_consensus.cc:504] T ceab6b0fccd54e909a64f66fe81a9a1f P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:00.092794 24468 raft_consensus.cc:1226] T ceab6b0fccd54e909a64f66fe81a9a1f P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20260925 03:04:00.093046 24662 raft_consensus.cc:348] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:00.094067 24662 raft_consensus.cc:374] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:04:00.094329 24662 raft_consensus.cc:729] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Initialized, Role: FOLLOWER
I20260925 03:04:00.094970 24662 consensus_queue.cc:260] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:00.097692 24662 ts_tablet_manager.cc:1428] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c: Time spent starting tablet: real 0.007s user 0.007s sys 0.001s
I20260925 03:04:00.099017 24662 tablet_bootstrap.cc:492] T 100c113a9fd84cc8adf2ec7b33ea0093 P eeab93abfdb141a895002ec9f312296c: Bootstrap starting.
I20260925 03:04:00.102219 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceab6b0fccd54e909a64f66fe81a9a1f" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:00.103880 25101 leader_election.cc:288] T ceab6b0fccd54e909a64f66fe81a9a1f P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:00.104317 24662 tablet_bootstrap.cc:654] T 100c113a9fd84cc8adf2ec7b33ea0093 P eeab93abfdb141a895002ec9f312296c: Neither blocks nor log segments found. Creating new log.
I20260925 03:04:00.107956 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceab6b0fccd54e909a64f66fe81a9a1f" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:00.108668 24542 raft_consensus.cc:3037] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:00.120751 24542 raft_consensus.cc:2454] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 2.
I20260925 03:04:00.120996 25025 consensus_queue.cc:1046] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 03:04:00.121868 25025 raft_consensus.cc:3032] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [term 1 LEADER]: Stepping down as leader of term 1
I20260925 03:04:00.122175 25025 raft_consensus.cc:729] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:00.122995 25025 consensus_queue.cc:260] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:00.124213 25025 raft_consensus.cc:3037] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:00.123407 24437 leader_election.cc:302] T ceab6b0fccd54e909a64f66fe81a9a1f P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:04:00.129935 25030 raft_consensus.cc:2781] T ceab6b0fccd54e909a64f66fe81a9a1f P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 03:04:00.130403 25030 raft_consensus.cc:686] T ceab6b0fccd54e909a64f66fe81a9a1f P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 LEADER]: Becoming Leader. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:04:00.131201 25030 consensus_queue.cc:237] T ceab6b0fccd54e909a64f66fe81a9a1f P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:00.135833 25084 consensus_queue.cc:1035] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260925 03:04:00.137827 24367 catalog_manager.cc:5495] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c reported cstate change: term changed from 0 to 1, leader changed from <none> to eeab93abfdb141a895002ec9f312296c (127.22.193.3). New cstate: current_term: 1 leader_uuid: "eeab93abfdb141a895002ec9f312296c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: HEALTHY } } }
I20260925 03:04:00.158929 24662 tablet_bootstrap.cc:492] T 100c113a9fd84cc8adf2ec7b33ea0093 P eeab93abfdb141a895002ec9f312296c: No bootstrap required, opened a new log
I20260925 03:04:00.159471 24662 ts_tablet_manager.cc:1397] T 100c113a9fd84cc8adf2ec7b33ea0093 P eeab93abfdb141a895002ec9f312296c: Time spent bootstrapping tablet: real 0.061s user 0.037s sys 0.010s
I20260925 03:04:00.161528 25101 raft_consensus.cc:482] T 100c113a9fd84cc8adf2ec7b33ea0093 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:00.162084 25101 raft_consensus.cc:504] T 100c113a9fd84cc8adf2ec7b33ea0093 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:00.164254 25101 leader_election.cc:288] T 100c113a9fd84cc8adf2ec7b33ea0093 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:00.162266 24662 raft_consensus.cc:348] T 100c113a9fd84cc8adf2ec7b33ea0093 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:00.165205 24614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "100c113a9fd84cc8adf2ec7b33ea0093" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:00.165282 24662 raft_consensus.cc:374] T 100c113a9fd84cc8adf2ec7b33ea0093 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:04:00.167488 24662 raft_consensus.cc:729] T 100c113a9fd84cc8adf2ec7b33ea0093 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Initialized, Role: FOLLOWER
I20260925 03:04:00.168097 24662 consensus_queue.cc:260] T 100c113a9fd84cc8adf2ec7b33ea0093 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:00.169013 24614 raft_consensus.cc:2454] T 100c113a9fd84cc8adf2ec7b33ea0093 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 0.
I20260925 03:04:00.170253 24662 ts_tablet_manager.cc:1428] T 100c113a9fd84cc8adf2ec7b33ea0093 P eeab93abfdb141a895002ec9f312296c: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20260925 03:04:00.171235 24662 tablet_bootstrap.cc:492] T 420163a6cdce412ebc21c25fe3835638 P eeab93abfdb141a895002ec9f312296c: Bootstrap starting.
I20260925 03:04:00.171855 24436 leader_election.cc:302] T 100c113a9fd84cc8adf2ec7b33ea0093 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:00.172629 25101 raft_consensus.cc:2781] T 100c113a9fd84cc8adf2ec7b33ea0093 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 03:04:00.172991 25101 raft_consensus.cc:482] T 100c113a9fd84cc8adf2ec7b33ea0093 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:00.173287 25101 raft_consensus.cc:3037] T 100c113a9fd84cc8adf2ec7b33ea0093 P 869b0a8575b54d08ae2817fad5dd42d5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:04:00.174290 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "100c113a9fd84cc8adf2ec7b33ea0093" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:00.174916 24542 raft_consensus.cc:2454] T 100c113a9fd84cc8adf2ec7b33ea0093 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 0.
I20260925 03:04:00.176566 24662 tablet_bootstrap.cc:654] T 420163a6cdce412ebc21c25fe3835638 P eeab93abfdb141a895002ec9f312296c: Neither blocks nor log segments found. Creating new log.
I20260925 03:04:00.178017 25089 consensus_queue.cc:1035] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 03:04:00.179983 24371 catalog_manager.cc:5495] T ceab6b0fccd54e909a64f66fe81a9a1f P 869b0a8575b54d08ae2817fad5dd42d5 reported cstate change: term changed from 1 to 2, leader changed from eeab93abfdb141a895002ec9f312296c (127.22.193.3) to 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1). New cstate: current_term: 2 leader_uuid: "869b0a8575b54d08ae2817fad5dd42d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: UNKNOWN } } }
I20260925 03:04:00.195290 25101 raft_consensus.cc:504] T 100c113a9fd84cc8adf2ec7b33ea0093 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:00.199481 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "100c113a9fd84cc8adf2ec7b33ea0093" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:00.200085 24617 raft_consensus.cc:3037] T 100c113a9fd84cc8adf2ec7b33ea0093 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:04:00.211253 24617 raft_consensus.cc:2454] T 100c113a9fd84cc8adf2ec7b33ea0093 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 1.
I20260925 03:04:00.212409 24436 leader_election.cc:302] T 100c113a9fd84cc8adf2ec7b33ea0093 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:00.213145 25026 raft_consensus.cc:2781] T 100c113a9fd84cc8adf2ec7b33ea0093 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:04:00.213660 25026 raft_consensus.cc:686] T 100c113a9fd84cc8adf2ec7b33ea0093 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 LEADER]: Becoming Leader. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:04:00.214434 25026 consensus_queue.cc:237] T 100c113a9fd84cc8adf2ec7b33ea0093 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:00.213277 25101 leader_election.cc:288] T 100c113a9fd84cc8adf2ec7b33ea0093 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 1 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:00.218827 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "100c113a9fd84cc8adf2ec7b33ea0093" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:00.219425 24542 raft_consensus.cc:3037] T 100c113a9fd84cc8adf2ec7b33ea0093 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:04:00.226233 24542 raft_consensus.cc:2454] T 100c113a9fd84cc8adf2ec7b33ea0093 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 1.
I20260925 03:04:00.252235 24662 tablet_bootstrap.cc:492] T 420163a6cdce412ebc21c25fe3835638 P eeab93abfdb141a895002ec9f312296c: No bootstrap required, opened a new log
I20260925 03:04:00.252715 24662 ts_tablet_manager.cc:1397] T 420163a6cdce412ebc21c25fe3835638 P eeab93abfdb141a895002ec9f312296c: Time spent bootstrapping tablet: real 0.082s user 0.028s sys 0.039s
I20260925 03:04:00.260679 24662 raft_consensus.cc:348] T 420163a6cdce412ebc21c25fe3835638 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:00.261420 24662 raft_consensus.cc:374] T 420163a6cdce412ebc21c25fe3835638 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:04:00.261668 24662 raft_consensus.cc:729] T 420163a6cdce412ebc21c25fe3835638 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Initialized, Role: FOLLOWER
I20260925 03:04:00.262305 24662 consensus_queue.cc:260] T 420163a6cdce412ebc21c25fe3835638 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:00.264814 24371 catalog_manager.cc:5495] T 100c113a9fd84cc8adf2ec7b33ea0093 P 869b0a8575b54d08ae2817fad5dd42d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1). New cstate: current_term: 1 leader_uuid: "869b0a8575b54d08ae2817fad5dd42d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: UNKNOWN } } }
I20260925 03:04:00.270045 24662 ts_tablet_manager.cc:1428] T 420163a6cdce412ebc21c25fe3835638 P eeab93abfdb141a895002ec9f312296c: Time spent starting tablet: real 0.017s user 0.005s sys 0.000s
I20260925 03:04:00.290388 25025 raft_consensus.cc:482] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:00.290843 25025 raft_consensus.cc:504] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:00.292529 25025 leader_election.cc:288] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:00.293431 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35992316cb8d4c028333388eff02ecde" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:00.293470 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35992316cb8d4c028333388eff02ecde" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:00.294950 24588 leader_election.cc:302] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:00.295631 25025 raft_consensus.cc:2726] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 03:04:00.319054 24617 raft_consensus.cc:3037] T 420163a6cdce412ebc21c25fe3835638 P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:04:00.325645 25101 consensus_queue.cc:1035] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 03:04:00.337937 25026 consensus_queue.cc:1035] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260925 03:04:00.340837 25026 consensus_queue.cc:1035] T cc2df40db08649c9a75e90000b878559 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 03:04:00.354586 25101 consensus_queue.cc:1035] T cc2df40db08649c9a75e90000b878559 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 03:04:00.450402 24614 raft_consensus.cc:3037] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:04:00.543901 25131 consensus_queue.cc:1035] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260925 03:04:00.559846 25131 consensus_queue.cc:1035] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 03:04:00.657732 25026 consensus_queue.cc:1035] T 100c113a9fd84cc8adf2ec7b33ea0093 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 03:04:00.667019 25030 consensus_queue.cc:1035] T 100c113a9fd84cc8adf2ec7b33ea0093 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 03:04:00.685101 24617 raft_consensus.cc:1261] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 1. (term mismatch)
I20260925 03:04:00.685523 24617 pending_rounds.cc:76] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c: Aborting all ops after (but not including) 0
I20260925 03:04:00.685763 24617 pending_rounds.cc:97] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20260925 03:04:00.686081 24617 raft_consensus.cc:2866] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260925 03:04:00.687517 25030 consensus_queue.cc:1035] T ceab6b0fccd54e909a64f66fe81a9a1f P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 03:04:00.695185 24544 raft_consensus.cc:1261] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260925 03:04:00.700075 25101 consensus_queue.cc:1035] T ceab6b0fccd54e909a64f66fe81a9a1f P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 03:04:02.187418 23300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 03:04:02.197136 25153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:04:02.199424 25154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:04:02.201319 25156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 03:04:02.215827 23300 server_base.cc:1034] running on GCE node
I20260925 03:04:02.216890 23300 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260925 03:04:02.217119 23300 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260925 03:04:02.217272 23300 hybrid_clock.cc:648] HybridClock initialized: now 1790305442217261 us; error 0 us; skew 500 ppm
I20260925 03:04:02.217958 23300 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 03:04:02.224627 23300 webserver.cc:460] Webserver started at http://127.22.193.4:45513/ using document root <none> and password file <none>
I20260925 03:04:02.225174 23300 fs_manager.cc:359] Metadata directory not provided
I20260925 03:04:02.225333 23300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 03:04:02.225580 23300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 03:04:02.227051 23300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklAj8hX/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790305368086980-23300-0/minicluster-data/ts-3-root/instance:
uuid: "1bc1f2e62ab44af995165bd5116a9d43"
format_stamp: "Formatted at 2026-09-25 03:04:02 on dist-test-slave-cswq"
I20260925 03:04:02.231930 23300 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.005s
I20260925 03:04:02.236081 25161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 03:04:02.236999 23300 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 03:04:02.237263 23300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklAj8hX/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790305368086980-23300-0/minicluster-data/ts-3-root
uuid: "1bc1f2e62ab44af995165bd5116a9d43"
format_stamp: "Formatted at 2026-09-25 03:04:02 on dist-test-slave-cswq"
I20260925 03:04:02.237514 23300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklAj8hX/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790305368086980-23300-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-tasklAj8hX/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790305368086980-23300-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-tasklAj8hX/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790305368086980-23300-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 03:04:02.268934 23300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 03:04:02.270241 23300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 03:04:02.271709 23300 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 03:04:02.274112 23300 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 03:04:02.274293 23300 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 03:04:02.274550 23300 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 03:04:02.274693 23300 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 03:04:02.323169 23300 rpc_server.cc:307] RPC server started. Bound to: 127.22.193.4:43979
I20260925 03:04:02.323292 25223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.193.4:43979 every 8 connection(s)
I20260925 03:04:02.327824 23300 auto_leader_rebalancer-test.cc:413] add a tserver: 1bc1f2e62ab44af995165bd5116a9d43
I20260925 03:04:02.340553 25224 heartbeater.cc:347] Connected to a master server at 127.22.193.62:40353
I20260925 03:04:02.341575 25224 heartbeater.cc:464] Registering TS with master...
I20260925 03:04:02.342351 25224 heartbeater.cc:511] Master 127.22.193.62:40353 requested a full tablet report, sending...
I20260925 03:04:02.344125 24367 ts_manager.cc:194] Registered new tserver with Master: 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:02.345609 24367 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57656
I20260925 03:04:02.692004 24614 consensus_queue.cc:237] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } attrs { replace: true } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } }
I20260925 03:04:02.700591 24544 raft_consensus.cc:1261] T 8ac466793d094325ace224708b2158ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 03:04:02.701545 24464 raft_consensus.cc:1261] T 8ac466793d094325ace224708b2158ed P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 03:04:02.702075 25230 consensus_queue.cc:1035] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260925 03:04:02.703040 25149 consensus_queue.cc:1035] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260925 03:04:02.717114 24585 consensus_peers.cc:487] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c -> Peer 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979): Couldn't send request to peer 1bc1f2e62ab44af995165bd5116a9d43. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8ac466793d094325ace224708b2158ed. This is attempt 1: this message will repeat every 5th retry.
I20260925 03:04:02.715451 25149 raft_consensus.cc:2932] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } attrs { replace: true } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } } }
I20260925 03:04:02.717219 24544 raft_consensus.cc:2932] T 8ac466793d094325ace224708b2158ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } attrs { replace: true } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } } }
I20260925 03:04:02.717876 24464 raft_consensus.cc:2932] T 8ac466793d094325ace224708b2158ed P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } attrs { replace: true } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } } }
I20260925 03:04:02.732800 24617 consensus_queue.cc:237] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } }
I20260925 03:04:02.734938 24367 catalog_manager.cc:5495] T 8ac466793d094325ace224708b2158ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 reported cstate change: config changed from index -1 to 2, NON_VOTER 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) added. New cstate: current_term: 1 leader_uuid: "eeab93abfdb141a895002ec9f312296c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } attrs { replace: true } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } } }
I20260925 03:04:02.749935 24544 raft_consensus.cc:1261] T 0e54ca1757ee4e42956a9cebbeca4f62 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260925 03:04:02.750331 24585 consensus_peers.cc:487] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c -> Peer 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979): Couldn't send request to peer 1bc1f2e62ab44af995165bd5116a9d43. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0e54ca1757ee4e42956a9cebbeca4f62. This is attempt 1: this message will repeat every 5th retry.
I20260925 03:04:02.750337 24464 raft_consensus.cc:1261] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 03:04:02.751645 25230 consensus_queue.cc:1035] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260925 03:04:02.752985 25236 consensus_queue.cc:1035] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260925 03:04:02.762642 25149 raft_consensus.cc:2932] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } } }
I20260925 03:04:02.766323 24544 raft_consensus.cc:2932] T 0e54ca1757ee4e42956a9cebbeca4f62 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } } }
I20260925 03:04:02.776674 24464 raft_consensus.cc:2932] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } } }
I20260925 03:04:02.797319 24371 catalog_manager.cc:5495] T 0e54ca1757ee4e42956a9cebbeca4f62 P a04a37fc4aa7460cbfdbeeb4e451e1f5 reported cstate change: config changed from index -1 to 2, NON_VOTER 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) added. New cstate: current_term: 1 leader_uuid: "eeab93abfdb141a895002ec9f312296c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } } }
I20260925 03:04:02.808058 24464 consensus_queue.cc:237] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } }
I20260925 03:04:02.822151 24544 raft_consensus.cc:1261] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 03:04:02.823850 25099 consensus_queue.cc:1035] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260925 03:04:02.831974 24617 raft_consensus.cc:1261] T 15861cd28e354598a3c6bfc4337c34bb P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 03:04:02.836885 25247 consensus_queue.cc:1035] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.005s
W20260925 03:04:02.845575 24436 consensus_peers.cc:487] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 -> Peer 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979): Couldn't send request to peer 1bc1f2e62ab44af995165bd5116a9d43. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 15861cd28e354598a3c6bfc4337c34bb. This is attempt 1: this message will repeat every 5th retry.
I20260925 03:04:02.847081 25247 raft_consensus.cc:2932] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } } }
I20260925 03:04:02.848920 24542 raft_consensus.cc:2932] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } } }
I20260925 03:04:02.875599 24369 catalog_manager.cc:5495] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 reported cstate change: config changed from index -1 to 2, NON_VOTER 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) added. New cstate: current_term: 1 leader_uuid: "869b0a8575b54d08ae2817fad5dd42d5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } } }
I20260925 03:04:02.885526 24614 raft_consensus.cc:2932] T 15861cd28e354598a3c6bfc4337c34bb P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } } }
I20260925 03:04:02.891481 24544 consensus_queue.cc:237] T a4abe0ea757e4886be4945c5001e529a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } attrs { replace: true } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } }
I20260925 03:04:02.915923 24617 raft_consensus.cc:1261] T a4abe0ea757e4886be4945c5001e529a P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 03:04:02.922890 24470 raft_consensus.cc:1261] T a4abe0ea757e4886be4945c5001e529a P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 03:04:02.924351 25089 consensus_queue.cc:1035] T a4abe0ea757e4886be4945c5001e529a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260925 03:04:02.929991 25089 consensus_queue.cc:1035] T a4abe0ea757e4886be4945c5001e529a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260925 03:04:02.937997 24511 consensus_peers.cc:487] T a4abe0ea757e4886be4945c5001e529a P a04a37fc4aa7460cbfdbeeb4e451e1f5 -> Peer 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979): Couldn't send request to peer 1bc1f2e62ab44af995165bd5116a9d43. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a4abe0ea757e4886be4945c5001e529a. This is attempt 1: this message will repeat every 5th retry.
I20260925 03:04:02.954890 25089 raft_consensus.cc:2932] T a4abe0ea757e4886be4945c5001e529a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } attrs { replace: true } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } } }
I20260925 03:04:02.958003 24617 raft_consensus.cc:2932] T a4abe0ea757e4886be4945c5001e529a P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } attrs { replace: true } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } } }
I20260925 03:04:02.958767 24468 raft_consensus.cc:2932] T a4abe0ea757e4886be4945c5001e529a P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } attrs { replace: true } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } } }
I20260925 03:04:02.980792 24371 catalog_manager.cc:5495] T a4abe0ea757e4886be4945c5001e529a P 869b0a8575b54d08ae2817fad5dd42d5 reported cstate change: config changed from index -1 to 2, NON_VOTER 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) added. New cstate: current_term: 1 leader_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } attrs { replace: true } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } } }
I20260925 03:04:03.206266 25257 ts_tablet_manager.cc:927] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43: Initiating tablet copy from peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187)
I20260925 03:04:03.219020 25257 tablet_copy_client.cc:316] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43: tablet copy: Beginning tablet copy session from remote peer at address 127.22.193.1:42187
I20260925 03:04:03.226167 25259 ts_tablet_manager.cc:927] T 8ac466793d094325ace224708b2158ed P 1bc1f2e62ab44af995165bd5116a9d43: Initiating tablet copy from peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:03.236388 25259 tablet_copy_client.cc:316] T 8ac466793d094325ace224708b2158ed P 1bc1f2e62ab44af995165bd5116a9d43: tablet copy: Beginning tablet copy session from remote peer at address 127.22.193.3:35105
I20260925 03:04:03.255841 24483 tablet_copy_service.cc:140] P 869b0a8575b54d08ae2817fad5dd42d5: Received BeginTabletCopySession request for tablet 15861cd28e354598a3c6bfc4337c34bb from peer 1bc1f2e62ab44af995165bd5116a9d43 ({username='slave'} at 127.0.0.1:38920)
I20260925 03:04:03.256412 24483 tablet_copy_service.cc:161] P 869b0a8575b54d08ae2817fad5dd42d5: Beginning new tablet copy session on tablet 15861cd28e354598a3c6bfc4337c34bb from peer 1bc1f2e62ab44af995165bd5116a9d43 at {username='slave'} at 127.0.0.1:38920: session id = 1bc1f2e62ab44af995165bd5116a9d43-15861cd28e354598a3c6bfc4337c34bb
I20260925 03:04:03.260841 24632 tablet_copy_service.cc:140] P eeab93abfdb141a895002ec9f312296c: Received BeginTabletCopySession request for tablet 8ac466793d094325ace224708b2158ed from peer 1bc1f2e62ab44af995165bd5116a9d43 ({username='slave'} at 127.0.0.1:34546)
I20260925 03:04:03.261359 24632 tablet_copy_service.cc:161] P eeab93abfdb141a895002ec9f312296c: Beginning new tablet copy session on tablet 8ac466793d094325ace224708b2158ed from peer 1bc1f2e62ab44af995165bd5116a9d43 at {username='slave'} at 127.0.0.1:34546: session id = 1bc1f2e62ab44af995165bd5116a9d43-8ac466793d094325ace224708b2158ed
I20260925 03:04:03.264009 24483 tablet_copy_source_session.cc:205] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 03:04:03.268167 25257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15861cd28e354598a3c6bfc4337c34bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:04:03.268077 24632 tablet_copy_source_session.cc:205] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 03:04:03.277484 25259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ac466793d094325ace224708b2158ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:04:03.286226 25257 tablet_copy_client.cc:791] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43: tablet copy: Starting download of 0 data blocks...
I20260925 03:04:03.286928 25257 tablet_copy_client.cc:655] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43: tablet copy: Starting download of 1 WAL segments...
I20260925 03:04:03.292732 25259 tablet_copy_client.cc:791] T 8ac466793d094325ace224708b2158ed P 1bc1f2e62ab44af995165bd5116a9d43: tablet copy: Starting download of 0 data blocks...
I20260925 03:04:03.293311 25259 tablet_copy_client.cc:655] T 8ac466793d094325ace224708b2158ed P 1bc1f2e62ab44af995165bd5116a9d43: tablet copy: Starting download of 1 WAL segments...
I20260925 03:04:03.295244 25257 tablet_copy_client.cc:531] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 03:04:03.298547 25259 tablet_copy_client.cc:531] T 8ac466793d094325ace224708b2158ed P 1bc1f2e62ab44af995165bd5116a9d43: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 03:04:03.306821 25257 tablet_bootstrap.cc:492] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43: Bootstrap starting.
I20260925 03:04:03.306846 25259 tablet_bootstrap.cc:492] T 8ac466793d094325ace224708b2158ed P 1bc1f2e62ab44af995165bd5116a9d43: Bootstrap starting.
I20260925 03:04:03.321637 25264 ts_tablet_manager.cc:927] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43: Initiating tablet copy from peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:03.328919 25264 tablet_copy_client.cc:316] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43: tablet copy: Beginning tablet copy session from remote peer at address 127.22.193.3:35105
I20260925 03:04:03.330549 24632 tablet_copy_service.cc:140] P eeab93abfdb141a895002ec9f312296c: Received BeginTabletCopySession request for tablet 0e54ca1757ee4e42956a9cebbeca4f62 from peer 1bc1f2e62ab44af995165bd5116a9d43 ({username='slave'} at 127.0.0.1:34546)
I20260925 03:04:03.331141 24632 tablet_copy_service.cc:161] P eeab93abfdb141a895002ec9f312296c: Beginning new tablet copy session on tablet 0e54ca1757ee4e42956a9cebbeca4f62 from peer 1bc1f2e62ab44af995165bd5116a9d43 at {username='slave'} at 127.0.0.1:34546: session id = 1bc1f2e62ab44af995165bd5116a9d43-0e54ca1757ee4e42956a9cebbeca4f62
I20260925 03:04:03.339812 25259 tablet_bootstrap.cc:492] T 8ac466793d094325ace224708b2158ed P 1bc1f2e62ab44af995165bd5116a9d43: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 03:04:03.340831 25259 tablet_bootstrap.cc:492] T 8ac466793d094325ace224708b2158ed P 1bc1f2e62ab44af995165bd5116a9d43: Bootstrap complete.
I20260925 03:04:03.341121 25257 tablet_bootstrap.cc:492] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 03:04:03.341686 25259 ts_tablet_manager.cc:1397] T 8ac466793d094325ace224708b2158ed P 1bc1f2e62ab44af995165bd5116a9d43: Time spent bootstrapping tablet: real 0.035s user 0.018s sys 0.008s
I20260925 03:04:03.342000 25257 tablet_bootstrap.cc:492] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43: Bootstrap complete.
I20260925 03:04:03.342759 25257 ts_tablet_manager.cc:1397] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43: Time spent bootstrapping tablet: real 0.036s user 0.028s sys 0.000s
I20260925 03:04:03.344630 25259 raft_consensus.cc:348] T 8ac466793d094325ace224708b2158ed P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } attrs { replace: true } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } }
I20260925 03:04:03.345577 25259 raft_consensus.cc:729] T 8ac466793d094325ace224708b2158ed P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1bc1f2e62ab44af995165bd5116a9d43, State: Initialized, Role: LEARNER
I20260925 03:04:03.346055 25257 raft_consensus.cc:348] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } }
I20260925 03:04:03.346751 25257 raft_consensus.cc:729] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1bc1f2e62ab44af995165bd5116a9d43, State: Initialized, Role: LEARNER
I20260925 03:04:03.346221 25259 consensus_queue.cc:260] T 8ac466793d094325ace224708b2158ed P 1bc1f2e62ab44af995165bd5116a9d43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } attrs { replace: true } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } }
I20260925 03:04:03.346832 24632 tablet_copy_source_session.cc:205] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 03:04:03.347369 25257 consensus_queue.cc:260] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } }
I20260925 03:04:03.348414 25224 heartbeater.cc:503] Master 127.22.193.62:40353 was elected leader, sending a full tablet report...
I20260925 03:04:03.350252 25259 ts_tablet_manager.cc:1428] T 8ac466793d094325ace224708b2158ed P 1bc1f2e62ab44af995165bd5116a9d43: Time spent starting tablet: real 0.008s user 0.000s sys 0.006s
I20260925 03:04:03.352046 24631 tablet_copy_service.cc:342] P eeab93abfdb141a895002ec9f312296c: Request end of tablet copy session 1bc1f2e62ab44af995165bd5116a9d43-8ac466793d094325ace224708b2158ed received from {username='slave'} at 127.0.0.1:34546
I20260925 03:04:03.352497 24631 tablet_copy_service.cc:433] P eeab93abfdb141a895002ec9f312296c: ending tablet copy session 1bc1f2e62ab44af995165bd5116a9d43-8ac466793d094325ace224708b2158ed on tablet 8ac466793d094325ace224708b2158ed with peer 1bc1f2e62ab44af995165bd5116a9d43
I20260925 03:04:03.354954 25264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e54ca1757ee4e42956a9cebbeca4f62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:04:03.359756 25257 ts_tablet_manager.cc:1428] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43: Time spent starting tablet: real 0.017s user 0.004s sys 0.009s
I20260925 03:04:03.361539 24483 tablet_copy_service.cc:342] P 869b0a8575b54d08ae2817fad5dd42d5: Request end of tablet copy session 1bc1f2e62ab44af995165bd5116a9d43-15861cd28e354598a3c6bfc4337c34bb received from {username='slave'} at 127.0.0.1:38920
I20260925 03:04:03.361970 24483 tablet_copy_service.cc:433] P 869b0a8575b54d08ae2817fad5dd42d5: ending tablet copy session 1bc1f2e62ab44af995165bd5116a9d43-15861cd28e354598a3c6bfc4337c34bb on tablet 15861cd28e354598a3c6bfc4337c34bb with peer 1bc1f2e62ab44af995165bd5116a9d43
I20260925 03:04:03.376770 25264 tablet_copy_client.cc:791] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43: tablet copy: Starting download of 0 data blocks...
I20260925 03:04:03.377535 25264 tablet_copy_client.cc:655] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43: tablet copy: Starting download of 1 WAL segments...
I20260925 03:04:03.382125 25264 tablet_copy_client.cc:531] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 03:04:03.391306 25264 tablet_bootstrap.cc:492] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43: Bootstrap starting.
I20260925 03:04:03.416754 25264 tablet_bootstrap.cc:492] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 03:04:03.417598 25264 tablet_bootstrap.cc:492] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43: Bootstrap complete.
I20260925 03:04:03.418222 25264 ts_tablet_manager.cc:1397] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43: Time spent bootstrapping tablet: real 0.027s user 0.021s sys 0.004s
I20260925 03:04:03.420598 25264 raft_consensus.cc:348] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } }
I20260925 03:04:03.421438 25264 raft_consensus.cc:729] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1bc1f2e62ab44af995165bd5116a9d43, State: Initialized, Role: LEARNER
I20260925 03:04:03.422046 25264 consensus_queue.cc:260] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } }
I20260925 03:04:03.424479 25264 ts_tablet_manager.cc:1428] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 03:04:03.426805 24632 tablet_copy_service.cc:342] P eeab93abfdb141a895002ec9f312296c: Request end of tablet copy session 1bc1f2e62ab44af995165bd5116a9d43-0e54ca1757ee4e42956a9cebbeca4f62 received from {username='slave'} at 127.0.0.1:34546
I20260925 03:04:03.427212 24632 tablet_copy_service.cc:433] P eeab93abfdb141a895002ec9f312296c: ending tablet copy session 1bc1f2e62ab44af995165bd5116a9d43-0e54ca1757ee4e42956a9cebbeca4f62 on tablet 0e54ca1757ee4e42956a9cebbeca4f62 with peer 1bc1f2e62ab44af995165bd5116a9d43
I20260925 03:04:03.471284 25264 ts_tablet_manager.cc:927] T a4abe0ea757e4886be4945c5001e529a P 1bc1f2e62ab44af995165bd5116a9d43: Initiating tablet copy from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:03.472723 25264 tablet_copy_client.cc:316] T a4abe0ea757e4886be4945c5001e529a P 1bc1f2e62ab44af995165bd5116a9d43: tablet copy: Beginning tablet copy session from remote peer at address 127.22.193.2:36389
I20260925 03:04:03.488329 24558 tablet_copy_service.cc:140] P a04a37fc4aa7460cbfdbeeb4e451e1f5: Received BeginTabletCopySession request for tablet a4abe0ea757e4886be4945c5001e529a from peer 1bc1f2e62ab44af995165bd5116a9d43 ({username='slave'} at 127.0.0.1:33068)
I20260925 03:04:03.489002 24558 tablet_copy_service.cc:161] P a04a37fc4aa7460cbfdbeeb4e451e1f5: Beginning new tablet copy session on tablet a4abe0ea757e4886be4945c5001e529a from peer 1bc1f2e62ab44af995165bd5116a9d43 at {username='slave'} at 127.0.0.1:33068: session id = 1bc1f2e62ab44af995165bd5116a9d43-a4abe0ea757e4886be4945c5001e529a
I20260925 03:04:03.497174 24558 tablet_copy_source_session.cc:205] T a4abe0ea757e4886be4945c5001e529a P a04a37fc4aa7460cbfdbeeb4e451e1f5: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 03:04:03.501523 25264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4abe0ea757e4886be4945c5001e529a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:04:03.521680 25264 tablet_copy_client.cc:791] T a4abe0ea757e4886be4945c5001e529a P 1bc1f2e62ab44af995165bd5116a9d43: tablet copy: Starting download of 0 data blocks...
I20260925 03:04:03.522549 25264 tablet_copy_client.cc:655] T a4abe0ea757e4886be4945c5001e529a P 1bc1f2e62ab44af995165bd5116a9d43: tablet copy: Starting download of 1 WAL segments...
I20260925 03:04:03.526510 25264 tablet_copy_client.cc:531] T a4abe0ea757e4886be4945c5001e529a P 1bc1f2e62ab44af995165bd5116a9d43: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 03:04:03.534922 25264 tablet_bootstrap.cc:492] T a4abe0ea757e4886be4945c5001e529a P 1bc1f2e62ab44af995165bd5116a9d43: Bootstrap starting.
I20260925 03:04:03.551854 25264 tablet_bootstrap.cc:492] T a4abe0ea757e4886be4945c5001e529a P 1bc1f2e62ab44af995165bd5116a9d43: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 03:04:03.552783 25264 tablet_bootstrap.cc:492] T a4abe0ea757e4886be4945c5001e529a P 1bc1f2e62ab44af995165bd5116a9d43: Bootstrap complete.
I20260925 03:04:03.553427 25264 ts_tablet_manager.cc:1397] T a4abe0ea757e4886be4945c5001e529a P 1bc1f2e62ab44af995165bd5116a9d43: Time spent bootstrapping tablet: real 0.019s user 0.011s sys 0.008s
I20260925 03:04:03.555686 25264 raft_consensus.cc:348] T a4abe0ea757e4886be4945c5001e529a P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } attrs { replace: true } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } }
I20260925 03:04:03.556476 25264 raft_consensus.cc:729] T a4abe0ea757e4886be4945c5001e529a P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1bc1f2e62ab44af995165bd5116a9d43, State: Initialized, Role: LEARNER
I20260925 03:04:03.557013 25264 consensus_queue.cc:260] T a4abe0ea757e4886be4945c5001e529a P 1bc1f2e62ab44af995165bd5116a9d43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } attrs { replace: true } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } }
I20260925 03:04:03.559314 25264 ts_tablet_manager.cc:1428] T a4abe0ea757e4886be4945c5001e529a P 1bc1f2e62ab44af995165bd5116a9d43: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 03:04:03.567945 24558 tablet_copy_service.cc:342] P a04a37fc4aa7460cbfdbeeb4e451e1f5: Request end of tablet copy session 1bc1f2e62ab44af995165bd5116a9d43-a4abe0ea757e4886be4945c5001e529a received from {username='slave'} at 127.0.0.1:33068
I20260925 03:04:03.568375 24558 tablet_copy_service.cc:433] P a04a37fc4aa7460cbfdbeeb4e451e1f5: ending tablet copy session 1bc1f2e62ab44af995165bd5116a9d43-a4abe0ea757e4886be4945c5001e529a on tablet a4abe0ea757e4886be4945c5001e529a with peer 1bc1f2e62ab44af995165bd5116a9d43
I20260925 03:04:03.665652 25199 raft_consensus.cc:1203] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260925 03:04:03.724634 25199 raft_consensus.cc:1203] T 8ac466793d094325ace224708b2158ed P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260925 03:04:03.903971 25199 raft_consensus.cc:1203] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260925 03:04:03.919096 25199 raft_consensus.cc:1203] T a4abe0ea757e4886be4945c5001e529a P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260925 03:04:04.126935 25247 raft_consensus.cc:1050] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5: attempting to promote NON_VOTER 1bc1f2e62ab44af995165bd5116a9d43 to VOTER
I20260925 03:04:04.129535 25247 consensus_queue.cc:237] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:04.135980 25199 raft_consensus.cc:1261] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 LEARNER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260925 03:04:04.136330 24614 raft_consensus.cc:1261] T 15861cd28e354598a3c6bfc4337c34bb P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260925 03:04:04.137578 25099 consensus_queue.cc:1035] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260925 03:04:04.139693 25274 raft_consensus.cc:1050] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c: attempting to promote NON_VOTER 1bc1f2e62ab44af995165bd5116a9d43 to VOTER
I20260925 03:04:04.140177 25099 consensus_queue.cc:1035] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260925 03:04:04.142438 25274 consensus_queue.cc:237] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } attrs { replace: true } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:04.147886 24543 raft_consensus.cc:1261] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260925 03:04:04.156750 25247 consensus_queue.cc:1035] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260925 03:04:04.162624 25198 raft_consensus.cc:1261] T 8ac466793d094325ace224708b2158ed P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 LEARNER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260925 03:04:04.170141 24544 raft_consensus.cc:1261] T 8ac466793d094325ace224708b2158ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260925 03:04:04.171602 24470 raft_consensus.cc:1261] T 8ac466793d094325ace224708b2158ed P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260925 03:04:04.171625 25274 consensus_queue.cc:1035] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20260925 03:04:04.174187 25230 consensus_queue.cc:1035] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260925 03:04:04.173677 25150 raft_consensus.cc:2932] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:04.175058 25236 consensus_queue.cc:1035] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260925 03:04:04.194716 25230 raft_consensus.cc:2932] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } attrs { replace: true } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:04.197011 24545 raft_consensus.cc:2932] T 8ac466793d094325ace224708b2158ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } attrs { replace: true } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:04.201392 25199 raft_consensus.cc:2932] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:04.205300 24542 raft_consensus.cc:2932] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:04.206009 24617 raft_consensus.cc:2932] T 15861cd28e354598a3c6bfc4337c34bb P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:04.212726 24367 catalog_manager.cc:5495] T 8ac466793d094325ace224708b2158ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 reported cstate change: config changed from index 2 to 3, 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "eeab93abfdb141a895002ec9f312296c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } attrs { replace: true } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:04.225036 24369 catalog_manager.cc:5495] T 15861cd28e354598a3c6bfc4337c34bb P eeab93abfdb141a895002ec9f312296c reported cstate change: config changed from index 2 to 3, 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "869b0a8575b54d08ae2817fad5dd42d5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:04.237298 24470 raft_consensus.cc:2932] T 8ac466793d094325ace224708b2158ed P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } attrs { replace: true } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:04.240455 25198 raft_consensus.cc:2932] T 8ac466793d094325ace224708b2158ed P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } attrs { replace: true } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:04.273011 24470 consensus_queue.cc:237] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:04.280848 25198 raft_consensus.cc:1261] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260925 03:04:04.283444 25150 consensus_queue.cc:1035] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260925 03:04:04.288157 24542 raft_consensus.cc:1261] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260925 03:04:04.290005 25286 consensus_queue.cc:1035] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260925 03:04:04.291755 25247 raft_consensus.cc:2932] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER eeab93abfdb141a895002ec9f312296c (127.22.193.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:04.302671 25198 raft_consensus.cc:2932] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER eeab93abfdb141a895002ec9f312296c (127.22.193.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:04.306533 24614 consensus_queue.cc:237] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:04.305979 24545 raft_consensus.cc:2932] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER eeab93abfdb141a895002ec9f312296c (127.22.193.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:04.309091 24358 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 15861cd28e354598a3c6bfc4337c34bb with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260925 03:04:04.328123 24547 raft_consensus.cc:1261] T 8ac466793d094325ace224708b2158ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260925 03:04:04.333170 25274 consensus_queue.cc:1035] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260925 03:04:04.331449 24368 catalog_manager.cc:5495] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43 reported cstate change: config changed from index 3 to 4, VOTER eeab93abfdb141a895002ec9f312296c (127.22.193.3) evicted. New cstate: current_term: 1 leader_uuid: "869b0a8575b54d08ae2817fad5dd42d5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:04.342991 25274 raft_consensus.cc:2932] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:04.337970 25198 raft_consensus.cc:1261] T 8ac466793d094325ace224708b2158ed P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260925 03:04:04.348405 25236 consensus_queue.cc:1035] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260925 03:04:04.355122 24545 raft_consensus.cc:2932] T 8ac466793d094325ace224708b2158ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:04.369980 24606 tablet_service.cc:1514] Processing DeleteTablet for tablet 15861cd28e354598a3c6bfc4337c34bb with delete_type TABLET_DATA_TOMBSTONED (TS eeab93abfdb141a895002ec9f312296c not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:38550
I20260925 03:04:04.372373 25291 tablet_replica.cc:330] T 15861cd28e354598a3c6bfc4337c34bb P eeab93abfdb141a895002ec9f312296c: stopping tablet replica
I20260925 03:04:04.373422 25291 raft_consensus.cc:2229] T 15861cd28e354598a3c6bfc4337c34bb P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 03:04:04.374114 25291 raft_consensus.cc:2258] T 15861cd28e354598a3c6bfc4337c34bb P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 03:04:04.359261 25198 raft_consensus.cc:2932] T 8ac466793d094325ace224708b2158ed P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:04.378079 25276 raft_consensus.cc:1050] T a4abe0ea757e4886be4945c5001e529a P a04a37fc4aa7460cbfdbeeb4e451e1f5: attempting to promote NON_VOTER 1bc1f2e62ab44af995165bd5116a9d43 to VOTER
I20260925 03:04:04.378258 25291 ts_tablet_manager.cc:1905] T 15861cd28e354598a3c6bfc4337c34bb P eeab93abfdb141a895002ec9f312296c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260925 03:04:04.380476 25276 consensus_queue.cc:237] T a4abe0ea757e4886be4945c5001e529a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } attrs { replace: true } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:04.383828 24355 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 8ac466793d094325ace224708b2158ed with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260925 03:04:04.394236 25291 ts_tablet_manager.cc:1918] T 15861cd28e354598a3c6bfc4337c34bb P eeab93abfdb141a895002ec9f312296c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260925 03:04:04.394811 25291 log.cc:1196] T 15861cd28e354598a3c6bfc4337c34bb P eeab93abfdb141a895002ec9f312296c: Deleting WAL directory at /tmp/dist-test-tasklAj8hX/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790305368086980-23300-0/minicluster-data/ts-2-root/wals/15861cd28e354598a3c6bfc4337c34bb
I20260925 03:04:04.397177 24355 catalog_manager.cc:4841] TS eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105): tablet 15861cd28e354598a3c6bfc4337c34bb (table test-workload [id=f20271173d58474897a63e58ef5a856d]) successfully deleted
I20260925 03:04:04.406468 25198 raft_consensus.cc:1261] T a4abe0ea757e4886be4945c5001e529a P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 LEARNER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260925 03:04:04.399435 24368 catalog_manager.cc:5495] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c reported cstate change: config changed from index 3 to 4, VOTER 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1) evicted. New cstate: current_term: 1 leader_uuid: "eeab93abfdb141a895002ec9f312296c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260925 03:04:04.410421 25276 consensus_queue.cc:1035] T a4abe0ea757e4886be4945c5001e529a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260925 03:04:04.412206 24614 raft_consensus.cc:1261] T a4abe0ea757e4886be4945c5001e529a P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260925 03:04:04.412418 24470 raft_consensus.cc:1261] T a4abe0ea757e4886be4945c5001e529a P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260925 03:04:04.413865 25276 consensus_queue.cc:1035] T a4abe0ea757e4886be4945c5001e529a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260925 03:04:04.417253 25276 consensus_queue.cc:1035] T a4abe0ea757e4886be4945c5001e529a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260925 03:04:04.428032 24456 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ac466793d094325ace224708b2158ed with delete_type TABLET_DATA_TOMBSTONED (TS 869b0a8575b54d08ae2817fad5dd42d5 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:55552
I20260925 03:04:04.439843 25252 raft_consensus.cc:2932] T a4abe0ea757e4886be4945c5001e529a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } attrs { replace: true } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:04.441818 25198 raft_consensus.cc:2932] T a4abe0ea757e4886be4945c5001e529a P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } attrs { replace: true } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:04.452775 25296 tablet_replica.cc:330] T 8ac466793d094325ace224708b2158ed P 869b0a8575b54d08ae2817fad5dd42d5: stopping tablet replica
I20260925 03:04:04.451992 24470 raft_consensus.cc:2932] T a4abe0ea757e4886be4945c5001e529a P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } attrs { replace: true } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:04.454295 25296 raft_consensus.cc:2229] T 8ac466793d094325ace224708b2158ed P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 03:04:04.456203 24368 catalog_manager.cc:5495] T a4abe0ea757e4886be4945c5001e529a P 1bc1f2e62ab44af995165bd5116a9d43 reported cstate change: config changed from index 2 to 3, 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } attrs { replace: true } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:04.457203 25296 raft_consensus.cc:2258] T 8ac466793d094325ace224708b2158ed P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 03:04:04.460536 25296 ts_tablet_manager.cc:1905] T 8ac466793d094325ace224708b2158ed P 869b0a8575b54d08ae2817fad5dd42d5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260925 03:04:04.471211 24614 raft_consensus.cc:2932] T a4abe0ea757e4886be4945c5001e529a P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } attrs { replace: true } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:04.490959 25296 ts_tablet_manager.cc:1918] T 8ac466793d094325ace224708b2158ed P 869b0a8575b54d08ae2817fad5dd42d5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260925 03:04:04.491423 25296 log.cc:1196] T 8ac466793d094325ace224708b2158ed P 869b0a8575b54d08ae2817fad5dd42d5: Deleting WAL directory at /tmp/dist-test-tasklAj8hX/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790305368086980-23300-0/minicluster-data/ts-0-root/wals/8ac466793d094325ace224708b2158ed
I20260925 03:04:04.493019 24358 catalog_manager.cc:4841] TS 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187): tablet 8ac466793d094325ace224708b2158ed (table test-workload [id=f20271173d58474897a63e58ef5a856d]) successfully deleted
I20260925 03:04:04.495298 24547 consensus_queue.cc:237] T a4abe0ea757e4886be4945c5001e529a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:04.500005 24614 raft_consensus.cc:1261] T a4abe0ea757e4886be4945c5001e529a P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260925 03:04:04.500026 25198 raft_consensus.cc:1261] T a4abe0ea757e4886be4945c5001e529a P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260925 03:04:04.501343 25252 consensus_queue.cc:1035] T a4abe0ea757e4886be4945c5001e529a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260925 03:04:04.502858 25276 consensus_queue.cc:1035] T a4abe0ea757e4886be4945c5001e529a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260925 03:04:04.515779 25230 raft_consensus.cc:1050] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c: attempting to promote NON_VOTER 1bc1f2e62ab44af995165bd5116a9d43 to VOTER
I20260925 03:04:04.517241 24617 raft_consensus.cc:2932] T a4abe0ea757e4886be4945c5001e529a P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:04.518321 25230 consensus_queue.cc:237] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:04.514534 25276 raft_consensus.cc:2932] T a4abe0ea757e4886be4945c5001e529a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:04.540504 25199 raft_consensus.cc:2932] T a4abe0ea757e4886be4945c5001e529a P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:04.541910 24547 raft_consensus.cc:1261] T 0e54ca1757ee4e42956a9cebbeca4f62 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260925 03:04:04.542865 24468 raft_consensus.cc:1261] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260925 03:04:04.544144 25274 consensus_queue.cc:1035] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260925 03:04:04.545931 25198 raft_consensus.cc:1261] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 LEARNER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260925 03:04:04.547760 25236 consensus_queue.cc:1035] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260925 03:04:04.549450 25236 consensus_queue.cc:1035] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260925 03:04:04.554360 24368 catalog_manager.cc:5495] T a4abe0ea757e4886be4945c5001e529a P eeab93abfdb141a895002ec9f312296c reported cstate change: config changed from index 3 to 4, VOTER 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1) evicted. New cstate: current_term: 1 leader_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:04.571023 24356 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet a4abe0ea757e4886be4945c5001e529a with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260925 03:04:04.594686 24456 tablet_service.cc:1514] Processing DeleteTablet for tablet a4abe0ea757e4886be4945c5001e529a with delete_type TABLET_DATA_TOMBSTONED (TS 869b0a8575b54d08ae2817fad5dd42d5 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:55552
I20260925 03:04:04.594141 25236 raft_consensus.cc:2932] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:04.595996 25296 tablet_replica.cc:330] T a4abe0ea757e4886be4945c5001e529a P 869b0a8575b54d08ae2817fad5dd42d5: stopping tablet replica
I20260925 03:04:04.597361 25296 raft_consensus.cc:2229] T a4abe0ea757e4886be4945c5001e529a P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 03:04:04.598150 25296 raft_consensus.cc:2258] T a4abe0ea757e4886be4945c5001e529a P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260925 03:04:04.602494 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 54505.3us
I20260925 03:04:04.603296 25296 ts_tablet_manager.cc:1905] T a4abe0ea757e4886be4945c5001e529a P 869b0a8575b54d08ae2817fad5dd42d5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260925 03:04:04.615125 24542 raft_consensus.cc:2932] T 0e54ca1757ee4e42956a9cebbeca4f62 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:04.638907 25199 raft_consensus.cc:2932] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:04.646334 24368 catalog_manager.cc:5495] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c reported cstate change: config changed from index 2 to 3, 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "eeab93abfdb141a895002ec9f312296c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260925 03:04:04.698378 24464 raft_consensus.cc:2932] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:04.707983 25296 ts_tablet_manager.cc:1918] T a4abe0ea757e4886be4945c5001e529a P 869b0a8575b54d08ae2817fad5dd42d5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260925 03:04:04.708451 25296 log.cc:1196] T a4abe0ea757e4886be4945c5001e529a P 869b0a8575b54d08ae2817fad5dd42d5: Deleting WAL directory at /tmp/dist-test-tasklAj8hX/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790305368086980-23300-0/minicluster-data/ts-0-root/wals/a4abe0ea757e4886be4945c5001e529a
I20260925 03:04:04.710985 24358 catalog_manager.cc:4841] TS 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187): tablet a4abe0ea757e4886be4945c5001e529a (table test-workload [id=f20271173d58474897a63e58ef5a856d]) successfully deleted
I20260925 03:04:04.738852 24622 consensus_queue.cc:237] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:04.747808 25199 raft_consensus.cc:1261] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260925 03:04:04.748318 24468 raft_consensus.cc:1261] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260925 03:04:04.749380 25274 consensus_queue.cc:1035] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260925 03:04:04.751221 25274 consensus_queue.cc:1035] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260925 03:04:04.757220 25236 raft_consensus.cc:2932] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:04.758440 25199 raft_consensus.cc:2932] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:04.762913 24468 raft_consensus.cc:2932] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:04.771036 24355 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 0e54ca1757ee4e42956a9cebbeca4f62 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260925 03:04:04.777963 24368 catalog_manager.cc:5495] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 reported cstate change: config changed from index 3 to 4, VOTER a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2) evicted. New cstate: current_term: 1 leader_uuid: "eeab93abfdb141a895002ec9f312296c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:04.789096 24534 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e54ca1757ee4e42956a9cebbeca4f62 with delete_type TABLET_DATA_TOMBSTONED (TS a04a37fc4aa7460cbfdbeeb4e451e1f5 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:36784
I20260925 03:04:04.790948 25305 tablet_replica.cc:330] T 0e54ca1757ee4e42956a9cebbeca4f62 P a04a37fc4aa7460cbfdbeeb4e451e1f5: stopping tablet replica
I20260925 03:04:04.791635 25305 raft_consensus.cc:2229] T 0e54ca1757ee4e42956a9cebbeca4f62 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 03:04:04.792209 25305 raft_consensus.cc:2258] T 0e54ca1757ee4e42956a9cebbeca4f62 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 03:04:04.794631 25305 ts_tablet_manager.cc:1905] T 0e54ca1757ee4e42956a9cebbeca4f62 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260925 03:04:04.806988 25305 ts_tablet_manager.cc:1918] T 0e54ca1757ee4e42956a9cebbeca4f62 P a04a37fc4aa7460cbfdbeeb4e451e1f5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260925 03:04:04.807340 25305 log.cc:1196] T 0e54ca1757ee4e42956a9cebbeca4f62 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Deleting WAL directory at /tmp/dist-test-tasklAj8hX/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790305368086980-23300-0/minicluster-data/ts-1-root/wals/0e54ca1757ee4e42956a9cebbeca4f62
I20260925 03:04:04.808969 24356 catalog_manager.cc:4841] TS a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389): tablet 0e54ca1757ee4e42956a9cebbeca4f62 (table test-workload [id=f20271173d58474897a63e58ef5a856d]) successfully deleted
I20260925 03:04:05.838513 24542 consensus_queue.cc:237] T b4ebd2b043024a548005c8b6b00b0449 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } }
I20260925 03:04:05.844084 24617 raft_consensus.cc:1261] T b4ebd2b043024a548005c8b6b00b0449 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 03:04:05.844193 24468 raft_consensus.cc:1261] T b4ebd2b043024a548005c8b6b00b0449 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 03:04:05.845456 25306 consensus_queue.cc:1035] T b4ebd2b043024a548005c8b6b00b0449 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260925 03:04:05.846539 25275 consensus_queue.cc:1035] T b4ebd2b043024a548005c8b6b00b0449 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260925 03:04:05.848930 24511 consensus_peers.cc:487] T b4ebd2b043024a548005c8b6b00b0449 P a04a37fc4aa7460cbfdbeeb4e451e1f5 -> Peer 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979): Couldn't send request to peer 1bc1f2e62ab44af995165bd5116a9d43. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b4ebd2b043024a548005c8b6b00b0449. This is attempt 1: this message will repeat every 5th retry.
I20260925 03:04:05.854205 25275 raft_consensus.cc:2932] T b4ebd2b043024a548005c8b6b00b0449 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } } }
I20260925 03:04:05.856472 24468 raft_consensus.cc:2932] T b4ebd2b043024a548005c8b6b00b0449 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } } }
I20260925 03:04:05.858845 24615 raft_consensus.cc:2932] T b4ebd2b043024a548005c8b6b00b0449 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } } }
I20260925 03:04:05.871624 24371 catalog_manager.cc:5495] T b4ebd2b043024a548005c8b6b00b0449 P 869b0a8575b54d08ae2817fad5dd42d5 reported cstate change: config changed from index -1 to 2, NON_VOTER 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) added. New cstate: current_term: 1 leader_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } } }
I20260925 03:04:05.883522 24615 consensus_queue.cc:237] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } }
I20260925 03:04:05.905061 24542 raft_consensus.cc:1261] T 6c81a1b3b616492ab6c964f45af36077 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260925 03:04:05.906248 24585 consensus_peers.cc:487] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c -> Peer 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979): Couldn't send request to peer 1bc1f2e62ab44af995165bd5116a9d43. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6c81a1b3b616492ab6c964f45af36077. This is attempt 1: this message will repeat every 5th retry.
I20260925 03:04:05.907475 25307 consensus_queue.cc:1035] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260925 03:04:05.918183 24468 raft_consensus.cc:1261] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 03:04:05.919553 25317 consensus_queue.cc:1035] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260925 03:04:05.921043 25307 raft_consensus.cc:2932] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } } }
I20260925 03:04:05.922612 24543 raft_consensus.cc:2932] T 6c81a1b3b616492ab6c964f45af36077 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } } }
I20260925 03:04:05.930744 24468 raft_consensus.cc:2932] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } } }
I20260925 03:04:05.938495 24368 catalog_manager.cc:5495] T 6c81a1b3b616492ab6c964f45af36077 P a04a37fc4aa7460cbfdbeeb4e451e1f5 reported cstate change: config changed from index -1 to 2, NON_VOTER 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) added. New cstate: current_term: 1 leader_uuid: "eeab93abfdb141a895002ec9f312296c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } } }
I20260925 03:04:05.949272 24468 consensus_queue.cc:237] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } }
I20260925 03:04:05.967348 24615 raft_consensus.cc:1261] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260925 03:04:05.967880 24436 consensus_peers.cc:487] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 -> Peer 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979): Couldn't send request to peer 1bc1f2e62ab44af995165bd5116a9d43. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c5aec38e66a947c681e6c2a72b05e6ca. This is attempt 1: this message will repeat every 5th retry.
I20260925 03:04:05.968294 24543 raft_consensus.cc:1261] T c5aec38e66a947c681e6c2a72b05e6ca P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 03:04:05.968756 25099 consensus_queue.cc:1035] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260925 03:04:05.969601 25286 consensus_queue.cc:1035] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260925 03:04:05.976958 25099 raft_consensus.cc:2932] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } } }
I20260925 03:04:05.979281 24615 raft_consensus.cc:2932] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } } }
I20260925 03:04:05.981253 24543 raft_consensus.cc:2932] T c5aec38e66a947c681e6c2a72b05e6ca P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } } }
I20260925 03:04:05.990410 24545 consensus_queue.cc:237] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } attrs { replace: true } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } }
I20260925 03:04:06.003696 24371 catalog_manager.cc:5495] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 reported cstate change: config changed from index -1 to 2, NON_VOTER 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) added. New cstate: current_term: 1 leader_uuid: "869b0a8575b54d08ae2817fad5dd42d5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260925 03:04:06.007522 24511 consensus_peers.cc:487] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 -> Peer 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979): Couldn't send request to peer 1bc1f2e62ab44af995165bd5116a9d43. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 71f461a6e2ff434993b78a2d077f0abd. This is attempt 1: this message will repeat every 5th retry.
I20260925 03:04:06.016806 24468 raft_consensus.cc:1261] T 71f461a6e2ff434993b78a2d077f0abd P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 03:04:06.018380 25275 consensus_queue.cc:1035] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260925 03:04:06.025482 24615 raft_consensus.cc:1261] T 71f461a6e2ff434993b78a2d077f0abd P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 03:04:06.028630 25306 consensus_queue.cc:1035] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260925 03:04:06.039691 25298 raft_consensus.cc:2932] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } attrs { replace: true } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } } }
I20260925 03:04:06.041095 24468 raft_consensus.cc:2932] T 71f461a6e2ff434993b78a2d077f0abd P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } attrs { replace: true } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } } }
I20260925 03:04:06.054761 24615 raft_consensus.cc:2932] T 71f461a6e2ff434993b78a2d077f0abd P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } attrs { replace: true } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } } }
I20260925 03:04:06.064703 24368 catalog_manager.cc:5495] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 reported cstate change: config changed from index -1 to 2, NON_VOTER 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) added. New cstate: current_term: 1 leader_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260925 03:04:06.445559 25327 ts_tablet_manager.cc:927] T b4ebd2b043024a548005c8b6b00b0449 P 1bc1f2e62ab44af995165bd5116a9d43: Initiating tablet copy from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:06.447129 25327 tablet_copy_client.cc:316] T b4ebd2b043024a548005c8b6b00b0449 P 1bc1f2e62ab44af995165bd5116a9d43: tablet copy: Beginning tablet copy session from remote peer at address 127.22.193.2:36389
I20260925 03:04:06.448789 24558 tablet_copy_service.cc:140] P a04a37fc4aa7460cbfdbeeb4e451e1f5: Received BeginTabletCopySession request for tablet b4ebd2b043024a548005c8b6b00b0449 from peer 1bc1f2e62ab44af995165bd5116a9d43 ({username='slave'} at 127.0.0.1:33068)
I20260925 03:04:06.449270 24558 tablet_copy_service.cc:161] P a04a37fc4aa7460cbfdbeeb4e451e1f5: Beginning new tablet copy session on tablet b4ebd2b043024a548005c8b6b00b0449 from peer 1bc1f2e62ab44af995165bd5116a9d43 at {username='slave'} at 127.0.0.1:33068: session id = 1bc1f2e62ab44af995165bd5116a9d43-b4ebd2b043024a548005c8b6b00b0449
I20260925 03:04:06.456223 24558 tablet_copy_source_session.cc:205] T b4ebd2b043024a548005c8b6b00b0449 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 03:04:06.462273 25327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4ebd2b043024a548005c8b6b00b0449. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:04:06.464100 25330 ts_tablet_manager.cc:927] T 71f461a6e2ff434993b78a2d077f0abd P 1bc1f2e62ab44af995165bd5116a9d43: Initiating tablet copy from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:06.467183 25331 ts_tablet_manager.cc:927] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43: Initiating tablet copy from peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:06.470686 25331 tablet_copy_client.cc:316] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43: tablet copy: Beginning tablet copy session from remote peer at address 127.22.193.3:35105
I20260925 03:04:06.472335 24632 tablet_copy_service.cc:140] P eeab93abfdb141a895002ec9f312296c: Received BeginTabletCopySession request for tablet 6c81a1b3b616492ab6c964f45af36077 from peer 1bc1f2e62ab44af995165bd5116a9d43 ({username='slave'} at 127.0.0.1:34546)
I20260925 03:04:06.472764 25330 tablet_copy_client.cc:316] T 71f461a6e2ff434993b78a2d077f0abd P 1bc1f2e62ab44af995165bd5116a9d43: tablet copy: Beginning tablet copy session from remote peer at address 127.22.193.2:36389
I20260925 03:04:06.476450 24558 tablet_copy_service.cc:140] P a04a37fc4aa7460cbfdbeeb4e451e1f5: Received BeginTabletCopySession request for tablet 71f461a6e2ff434993b78a2d077f0abd from peer 1bc1f2e62ab44af995165bd5116a9d43 ({username='slave'} at 127.0.0.1:33068)
I20260925 03:04:06.476963 24558 tablet_copy_service.cc:161] P a04a37fc4aa7460cbfdbeeb4e451e1f5: Beginning new tablet copy session on tablet 71f461a6e2ff434993b78a2d077f0abd from peer 1bc1f2e62ab44af995165bd5116a9d43 at {username='slave'} at 127.0.0.1:33068: session id = 1bc1f2e62ab44af995165bd5116a9d43-71f461a6e2ff434993b78a2d077f0abd
I20260925 03:04:06.480050 25327 tablet_copy_client.cc:791] T b4ebd2b043024a548005c8b6b00b0449 P 1bc1f2e62ab44af995165bd5116a9d43: tablet copy: Starting download of 0 data blocks...
I20260925 03:04:06.480660 25327 tablet_copy_client.cc:655] T b4ebd2b043024a548005c8b6b00b0449 P 1bc1f2e62ab44af995165bd5116a9d43: tablet copy: Starting download of 1 WAL segments...
I20260925 03:04:06.472867 24632 tablet_copy_service.cc:161] P eeab93abfdb141a895002ec9f312296c: Beginning new tablet copy session on tablet 6c81a1b3b616492ab6c964f45af36077 from peer 1bc1f2e62ab44af995165bd5116a9d43 at {username='slave'} at 127.0.0.1:34546: session id = 1bc1f2e62ab44af995165bd5116a9d43-6c81a1b3b616492ab6c964f45af36077
I20260925 03:04:06.488452 25327 tablet_copy_client.cc:531] T b4ebd2b043024a548005c8b6b00b0449 P 1bc1f2e62ab44af995165bd5116a9d43: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 03:04:06.491039 24632 tablet_copy_source_session.cc:205] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 03:04:06.492318 24558 tablet_copy_source_session.cc:205] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 03:04:06.494287 25331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c81a1b3b616492ab6c964f45af36077. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:04:06.497135 25330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71f461a6e2ff434993b78a2d077f0abd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:04:06.500896 25327 tablet_bootstrap.cc:492] T b4ebd2b043024a548005c8b6b00b0449 P 1bc1f2e62ab44af995165bd5116a9d43: Bootstrap starting.
I20260925 03:04:06.519325 25331 tablet_copy_client.cc:791] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43: tablet copy: Starting download of 0 data blocks...
I20260925 03:04:06.519973 25331 tablet_copy_client.cc:655] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43: tablet copy: Starting download of 1 WAL segments...
I20260925 03:04:06.523545 25331 tablet_copy_client.cc:531] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 03:04:06.525246 25330 tablet_copy_client.cc:791] T 71f461a6e2ff434993b78a2d077f0abd P 1bc1f2e62ab44af995165bd5116a9d43: tablet copy: Starting download of 0 data blocks...
I20260925 03:04:06.525879 25330 tablet_copy_client.cc:655] T 71f461a6e2ff434993b78a2d077f0abd P 1bc1f2e62ab44af995165bd5116a9d43: tablet copy: Starting download of 1 WAL segments...
I20260925 03:04:06.529520 25330 tablet_copy_client.cc:531] T 71f461a6e2ff434993b78a2d077f0abd P 1bc1f2e62ab44af995165bd5116a9d43: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 03:04:06.534381 25331 tablet_bootstrap.cc:492] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43: Bootstrap starting.
I20260925 03:04:06.539664 25330 tablet_bootstrap.cc:492] T 71f461a6e2ff434993b78a2d077f0abd P 1bc1f2e62ab44af995165bd5116a9d43: Bootstrap starting.
I20260925 03:04:06.572156 25327 tablet_bootstrap.cc:492] T b4ebd2b043024a548005c8b6b00b0449 P 1bc1f2e62ab44af995165bd5116a9d43: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 03:04:06.580011 25336 ts_tablet_manager.cc:927] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43: Initiating tablet copy from peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187)
I20260925 03:04:06.575037 25327 tablet_bootstrap.cc:492] T b4ebd2b043024a548005c8b6b00b0449 P 1bc1f2e62ab44af995165bd5116a9d43: Bootstrap complete.
I20260925 03:04:06.592126 25336 tablet_copy_client.cc:316] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43: tablet copy: Beginning tablet copy session from remote peer at address 127.22.193.1:42187
I20260925 03:04:06.594067 25327 ts_tablet_manager.cc:1397] T b4ebd2b043024a548005c8b6b00b0449 P 1bc1f2e62ab44af995165bd5116a9d43: Time spent bootstrapping tablet: real 0.093s user 0.018s sys 0.008s
I20260925 03:04:06.594511 24483 tablet_copy_service.cc:140] P 869b0a8575b54d08ae2817fad5dd42d5: Received BeginTabletCopySession request for tablet c5aec38e66a947c681e6c2a72b05e6ca from peer 1bc1f2e62ab44af995165bd5116a9d43 ({username='slave'} at 127.0.0.1:38920)
I20260925 03:04:06.595094 24483 tablet_copy_service.cc:161] P 869b0a8575b54d08ae2817fad5dd42d5: Beginning new tablet copy session on tablet c5aec38e66a947c681e6c2a72b05e6ca from peer 1bc1f2e62ab44af995165bd5116a9d43 at {username='slave'} at 127.0.0.1:38920: session id = 1bc1f2e62ab44af995165bd5116a9d43-c5aec38e66a947c681e6c2a72b05e6ca
I20260925 03:04:06.596877 25327 raft_consensus.cc:348] T b4ebd2b043024a548005c8b6b00b0449 P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } }
I20260925 03:04:06.597725 25327 raft_consensus.cc:729] T b4ebd2b043024a548005c8b6b00b0449 P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1bc1f2e62ab44af995165bd5116a9d43, State: Initialized, Role: LEARNER
I20260925 03:04:06.598366 25327 consensus_queue.cc:260] T b4ebd2b043024a548005c8b6b00b0449 P 1bc1f2e62ab44af995165bd5116a9d43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } }
I20260925 03:04:06.606091 25330 tablet_bootstrap.cc:492] T 71f461a6e2ff434993b78a2d077f0abd P 1bc1f2e62ab44af995165bd5116a9d43: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 03:04:06.606932 25330 tablet_bootstrap.cc:492] T 71f461a6e2ff434993b78a2d077f0abd P 1bc1f2e62ab44af995165bd5116a9d43: Bootstrap complete.
I20260925 03:04:06.607187 25331 tablet_bootstrap.cc:492] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 03:04:06.607487 25330 ts_tablet_manager.cc:1397] T 71f461a6e2ff434993b78a2d077f0abd P 1bc1f2e62ab44af995165bd5116a9d43: Time spent bootstrapping tablet: real 0.068s user 0.021s sys 0.011s
I20260925 03:04:06.608032 25331 tablet_bootstrap.cc:492] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43: Bootstrap complete.
I20260925 03:04:06.608739 25331 ts_tablet_manager.cc:1397] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43: Time spent bootstrapping tablet: real 0.075s user 0.008s sys 0.019s
I20260925 03:04:06.611407 25331 raft_consensus.cc:348] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } }
I20260925 03:04:06.612035 24483 tablet_copy_source_session.cc:205] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 03:04:06.612263 25331 raft_consensus.cc:729] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1bc1f2e62ab44af995165bd5116a9d43, State: Initialized, Role: LEARNER
I20260925 03:04:06.613046 25331 consensus_queue.cc:260] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } }
I20260925 03:04:06.614897 25327 ts_tablet_manager.cc:1428] T b4ebd2b043024a548005c8b6b00b0449 P 1bc1f2e62ab44af995165bd5116a9d43: Time spent starting tablet: real 0.020s user 0.006s sys 0.000s
I20260925 03:04:06.615671 25336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5aec38e66a947c681e6c2a72b05e6ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:04:06.616072 25330 raft_consensus.cc:348] T 71f461a6e2ff434993b78a2d077f0abd P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } attrs { replace: true } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } }
I20260925 03:04:06.617057 25330 raft_consensus.cc:729] T 71f461a6e2ff434993b78a2d077f0abd P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1bc1f2e62ab44af995165bd5116a9d43, State: Initialized, Role: LEARNER
I20260925 03:04:06.617743 25330 consensus_queue.cc:260] T 71f461a6e2ff434993b78a2d077f0abd P 1bc1f2e62ab44af995165bd5116a9d43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } attrs { replace: true } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } }
I20260925 03:04:06.622406 24558 tablet_copy_service.cc:342] P a04a37fc4aa7460cbfdbeeb4e451e1f5: Request end of tablet copy session 1bc1f2e62ab44af995165bd5116a9d43-b4ebd2b043024a548005c8b6b00b0449 received from {username='slave'} at 127.0.0.1:33068
I20260925 03:04:06.622848 24558 tablet_copy_service.cc:433] P a04a37fc4aa7460cbfdbeeb4e451e1f5: ending tablet copy session 1bc1f2e62ab44af995165bd5116a9d43-b4ebd2b043024a548005c8b6b00b0449 on tablet b4ebd2b043024a548005c8b6b00b0449 with peer 1bc1f2e62ab44af995165bd5116a9d43
I20260925 03:04:06.631018 25331 ts_tablet_manager.cc:1428] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43: Time spent starting tablet: real 0.022s user 0.014s sys 0.005s
I20260925 03:04:06.637753 24632 tablet_copy_service.cc:342] P eeab93abfdb141a895002ec9f312296c: Request end of tablet copy session 1bc1f2e62ab44af995165bd5116a9d43-6c81a1b3b616492ab6c964f45af36077 received from {username='slave'} at 127.0.0.1:34546
I20260925 03:04:06.638365 24632 tablet_copy_service.cc:433] P eeab93abfdb141a895002ec9f312296c: ending tablet copy session 1bc1f2e62ab44af995165bd5116a9d43-6c81a1b3b616492ab6c964f45af36077 on tablet 6c81a1b3b616492ab6c964f45af36077 with peer 1bc1f2e62ab44af995165bd5116a9d43
I20260925 03:04:06.643522 25336 tablet_copy_client.cc:791] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43: tablet copy: Starting download of 0 data blocks...
I20260925 03:04:06.644145 25336 tablet_copy_client.cc:655] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43: tablet copy: Starting download of 1 WAL segments...
I20260925 03:04:06.647948 25330 ts_tablet_manager.cc:1428] T 71f461a6e2ff434993b78a2d077f0abd P 1bc1f2e62ab44af995165bd5116a9d43: Time spent starting tablet: real 0.040s user 0.010s sys 0.000s
I20260925 03:04:06.649979 24558 tablet_copy_service.cc:342] P a04a37fc4aa7460cbfdbeeb4e451e1f5: Request end of tablet copy session 1bc1f2e62ab44af995165bd5116a9d43-71f461a6e2ff434993b78a2d077f0abd received from {username='slave'} at 127.0.0.1:33068
I20260925 03:04:06.650398 24558 tablet_copy_service.cc:433] P a04a37fc4aa7460cbfdbeeb4e451e1f5: ending tablet copy session 1bc1f2e62ab44af995165bd5116a9d43-71f461a6e2ff434993b78a2d077f0abd on tablet 71f461a6e2ff434993b78a2d077f0abd with peer 1bc1f2e62ab44af995165bd5116a9d43
I20260925 03:04:06.651639 25336 tablet_copy_client.cc:531] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 03:04:06.664116 25336 tablet_bootstrap.cc:492] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43: Bootstrap starting.
I20260925 03:04:06.688102 25336 tablet_bootstrap.cc:492] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 03:04:06.689088 25336 tablet_bootstrap.cc:492] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43: Bootstrap complete.
I20260925 03:04:06.689700 25336 ts_tablet_manager.cc:1397] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43: Time spent bootstrapping tablet: real 0.026s user 0.022s sys 0.004s
I20260925 03:04:06.692396 25336 raft_consensus.cc:348] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } }
I20260925 03:04:06.693547 25336 raft_consensus.cc:729] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1bc1f2e62ab44af995165bd5116a9d43, State: Initialized, Role: LEARNER
I20260925 03:04:06.694576 25336 consensus_queue.cc:260] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } }
I20260925 03:04:06.698449 25336 ts_tablet_manager.cc:1428] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260925 03:04:06.700378 24483 tablet_copy_service.cc:342] P 869b0a8575b54d08ae2817fad5dd42d5: Request end of tablet copy session 1bc1f2e62ab44af995165bd5116a9d43-c5aec38e66a947c681e6c2a72b05e6ca received from {username='slave'} at 127.0.0.1:38920
I20260925 03:04:06.700812 24483 tablet_copy_service.cc:433] P 869b0a8575b54d08ae2817fad5dd42d5: ending tablet copy session 1bc1f2e62ab44af995165bd5116a9d43-c5aec38e66a947c681e6c2a72b05e6ca on tablet c5aec38e66a947c681e6c2a72b05e6ca with peer 1bc1f2e62ab44af995165bd5116a9d43
I20260925 03:04:06.886646 25198 raft_consensus.cc:1203] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260925 03:04:06.938081 25198 raft_consensus.cc:1203] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260925 03:04:06.963352 25198 raft_consensus.cc:1203] T 71f461a6e2ff434993b78a2d077f0abd P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260925 03:04:06.977402 25198 raft_consensus.cc:1203] T b4ebd2b043024a548005c8b6b00b0449 P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260925 03:04:07.285413 25307 raft_consensus.cc:1050] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c: attempting to promote NON_VOTER 1bc1f2e62ab44af995165bd5116a9d43 to VOTER
I20260925 03:04:07.287976 25307 consensus_queue.cc:237] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:07.295270 25198 raft_consensus.cc:1261] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 LEARNER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260925 03:04:07.295379 24543 raft_consensus.cc:1261] T 6c81a1b3b616492ab6c964f45af36077 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260925 03:04:07.296769 24468 raft_consensus.cc:1261] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260925 03:04:07.296924 25343 consensus_queue.cc:1035] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260925 03:04:07.298403 25329 consensus_queue.cc:1035] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260925 03:04:07.300212 25343 consensus_queue.cc:1035] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260925 03:04:07.318002 25343 raft_consensus.cc:2932] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:07.319012 24468 raft_consensus.cc:2932] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:07.323069 25198 raft_consensus.cc:2932] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:07.330325 24542 raft_consensus.cc:2932] T 6c81a1b3b616492ab6c964f45af36077 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:07.338236 24371 catalog_manager.cc:5495] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c reported cstate change: config changed from index 2 to 3, 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "eeab93abfdb141a895002ec9f312296c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260925 03:04:07.378333 24617 consensus_queue.cc:237] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:07.386358 25198 raft_consensus.cc:1261] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260925 03:04:07.387987 25307 consensus_queue.cc:1035] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260925 03:04:07.386489 24468 raft_consensus.cc:1261] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260925 03:04:07.390420 25275 raft_consensus.cc:1050] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5: attempting to promote NON_VOTER 1bc1f2e62ab44af995165bd5116a9d43 to VOTER
I20260925 03:04:07.392005 25307 consensus_queue.cc:1035] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260925 03:04:07.392637 25275 consensus_queue.cc:237] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } attrs { replace: true } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:07.401165 24468 raft_consensus.cc:2932] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:07.398988 25307 raft_consensus.cc:2932] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:07.412322 24355 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 6c81a1b3b616492ab6c964f45af36077 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260925 03:04:07.414377 25198 raft_consensus.cc:2932] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:07.422531 24369 catalog_manager.cc:5495] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 reported cstate change: config changed from index 3 to 4, VOTER a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2) evicted. New cstate: current_term: 1 leader_uuid: "eeab93abfdb141a895002ec9f312296c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:07.432976 25199 raft_consensus.cc:1261] T 71f461a6e2ff434993b78a2d077f0abd P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 LEARNER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260925 03:04:07.433318 24617 raft_consensus.cc:1261] T 71f461a6e2ff434993b78a2d077f0abd P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260925 03:04:07.433449 24468 raft_consensus.cc:1261] T 71f461a6e2ff434993b78a2d077f0abd P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260925 03:04:07.434867 25275 consensus_queue.cc:1035] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260925 03:04:07.436057 24534 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c81a1b3b616492ab6c964f45af36077 with delete_type TABLET_DATA_TOMBSTONED (TS a04a37fc4aa7460cbfdbeeb4e451e1f5 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:36784
I20260925 03:04:07.436213 25311 consensus_queue.cc:1035] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260925 03:04:07.438145 25311 consensus_queue.cc:1035] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260925 03:04:07.440884 25355 tablet_replica.cc:330] T 6c81a1b3b616492ab6c964f45af36077 P a04a37fc4aa7460cbfdbeeb4e451e1f5: stopping tablet replica
I20260925 03:04:07.458330 25306 raft_consensus.cc:2932] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } attrs { replace: true } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:07.472050 25323 raft_consensus.cc:1050] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5: attempting to promote NON_VOTER 1bc1f2e62ab44af995165bd5116a9d43 to VOTER
I20260925 03:04:07.471284 24468 raft_consensus.cc:2932] T 71f461a6e2ff434993b78a2d077f0abd P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } attrs { replace: true } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:07.474217 25199 raft_consensus.cc:2932] T 71f461a6e2ff434993b78a2d077f0abd P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } attrs { replace: true } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:07.475093 25323 consensus_queue.cc:237] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:07.477617 24369 catalog_manager.cc:5495] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 reported cstate change: config changed from index 2 to 3, 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260925 03:04:07.466567 25355 raft_consensus.cc:2229] T 6c81a1b3b616492ab6c964f45af36077 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 03:04:07.497360 25355 raft_consensus.cc:2258] T 6c81a1b3b616492ab6c964f45af36077 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 03:04:07.497146 24617 raft_consensus.cc:2932] T 71f461a6e2ff434993b78a2d077f0abd P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } attrs { replace: true } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:07.510253 25355 ts_tablet_manager.cc:1905] T 6c81a1b3b616492ab6c964f45af36077 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260925 03:04:07.510365 25198 raft_consensus.cc:1261] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 LEARNER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260925 03:04:07.511780 25286 consensus_queue.cc:1035] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260925 03:04:07.522107 24542 consensus_queue.cc:237] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:07.526412 24622 raft_consensus.cc:1261] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260925 03:04:07.527999 25323 consensus_queue.cc:1035] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260925 03:04:07.531497 25355 ts_tablet_manager.cc:1918] T 6c81a1b3b616492ab6c964f45af36077 P a04a37fc4aa7460cbfdbeeb4e451e1f5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260925 03:04:07.533874 25355 log.cc:1196] T 6c81a1b3b616492ab6c964f45af36077 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Deleting WAL directory at /tmp/dist-test-tasklAj8hX/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790305368086980-23300-0/minicluster-data/ts-1-root/wals/6c81a1b3b616492ab6c964f45af36077
I20260925 03:04:07.535543 24622 raft_consensus.cc:1261] T 71f461a6e2ff434993b78a2d077f0abd P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260925 03:04:07.535903 24356 catalog_manager.cc:4841] TS a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389): tablet 6c81a1b3b616492ab6c964f45af36077 (table test-workload [id=f20271173d58474897a63e58ef5a856d]) successfully deleted
I20260925 03:04:07.537163 25275 consensus_queue.cc:1035] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260925 03:04:07.540472 25198 raft_consensus.cc:1261] T 71f461a6e2ff434993b78a2d077f0abd P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260925 03:04:07.544126 25353 consensus_queue.cc:1035] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260925 03:04:07.546612 25286 raft_consensus.cc:2932] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:07.548388 25311 raft_consensus.cc:2932] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:07.551301 24543 raft_consensus.cc:1261] T c5aec38e66a947c681e6c2a72b05e6ca P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260925 03:04:07.560318 24622 raft_consensus.cc:2932] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:07.561159 24614 raft_consensus.cc:2932] T 71f461a6e2ff434993b78a2d077f0abd P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:07.565157 25275 raft_consensus.cc:1050] T b4ebd2b043024a548005c8b6b00b0449 P a04a37fc4aa7460cbfdbeeb4e451e1f5: attempting to promote NON_VOTER 1bc1f2e62ab44af995165bd5116a9d43 to VOTER
I20260925 03:04:07.567412 25275 consensus_queue.cc:237] T b4ebd2b043024a548005c8b6b00b0449 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:07.571381 25199 raft_consensus.cc:2932] T 71f461a6e2ff434993b78a2d077f0abd P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:07.574374 25099 consensus_queue.cc:1035] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260925 03:04:07.584870 25198 raft_consensus.cc:2932] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:07.587394 24368 catalog_manager.cc:5495] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c reported cstate change: config changed from index 2 to 3, 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "869b0a8575b54d08ae2817fad5dd42d5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:07.590909 24356 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 71f461a6e2ff434993b78a2d077f0abd with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260925 03:04:07.605221 24369 catalog_manager.cc:5495] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 reported cstate change: config changed from index 3 to 4, VOTER 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1) evicted. New cstate: current_term: 1 leader_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260925 03:04:07.606424 24622 raft_consensus.cc:1261] T b4ebd2b043024a548005c8b6b00b0449 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260925 03:04:07.607033 25197 raft_consensus.cc:1261] T b4ebd2b043024a548005c8b6b00b0449 P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 LEARNER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260925 03:04:07.611701 25353 consensus_queue.cc:1035] T b4ebd2b043024a548005c8b6b00b0449 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260925 03:04:07.613039 25311 consensus_queue.cc:1035] T b4ebd2b043024a548005c8b6b00b0449 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260925 03:04:07.625746 24468 raft_consensus.cc:1261] T b4ebd2b043024a548005c8b6b00b0449 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260925 03:04:07.633688 24467 consensus_queue.cc:237] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:07.638454 24456 tablet_service.cc:1514] Processing DeleteTablet for tablet 71f461a6e2ff434993b78a2d077f0abd with delete_type TABLET_DATA_TOMBSTONED (TS 869b0a8575b54d08ae2817fad5dd42d5 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:55552
I20260925 03:04:07.642338 25369 tablet_replica.cc:330] T 71f461a6e2ff434993b78a2d077f0abd P 869b0a8575b54d08ae2817fad5dd42d5: stopping tablet replica
I20260925 03:04:07.644136 24614 raft_consensus.cc:1261] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260925 03:04:07.646166 25197 raft_consensus.cc:1261] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260925 03:04:07.647004 25363 consensus_queue.cc:1035] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260925 03:04:07.647500 24545 raft_consensus.cc:2932] T c5aec38e66a947c681e6c2a72b05e6ca P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:07.669271 25275 consensus_queue.cc:1035] T b4ebd2b043024a548005c8b6b00b0449 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260925 03:04:07.675203 25099 consensus_queue.cc:1035] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260925 03:04:07.677927 25306 raft_consensus.cc:2932] T b4ebd2b043024a548005c8b6b00b0449 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:07.690924 25364 raft_consensus.cc:2932] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:07.701958 24358 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet c5aec38e66a947c681e6c2a72b05e6ca with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260925 03:04:07.704152 25199 raft_consensus.cc:2932] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:07.706523 24371 catalog_manager.cc:5495] T b4ebd2b043024a548005c8b6b00b0449 P a04a37fc4aa7460cbfdbeeb4e451e1f5 reported cstate change: config changed from index 2 to 3, 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260925 03:04:07.717612 24615 raft_consensus.cc:2932] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:07.720028 25197 raft_consensus.cc:2932] T b4ebd2b043024a548005c8b6b00b0449 P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:07.727727 24622 raft_consensus.cc:2932] T b4ebd2b043024a548005c8b6b00b0449 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:07.729442 24369 catalog_manager.cc:5495] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 reported cstate change: config changed from index 3 to 4, VOTER a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2) evicted. New cstate: current_term: 1 leader_uuid: "869b0a8575b54d08ae2817fad5dd42d5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260925 03:04:07.750695 25369 raft_consensus.cc:2229] T 71f461a6e2ff434993b78a2d077f0abd P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 03:04:07.751483 25369 raft_consensus.cc:2258] T 71f461a6e2ff434993b78a2d077f0abd P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 03:04:07.755419 25369 ts_tablet_manager.cc:1905] T 71f461a6e2ff434993b78a2d077f0abd P 869b0a8575b54d08ae2817fad5dd42d5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260925 03:04:07.782007 24467 raft_consensus.cc:2932] T b4ebd2b043024a548005c8b6b00b0449 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:07.795535 24534 tablet_service.cc:1514] Processing DeleteTablet for tablet c5aec38e66a947c681e6c2a72b05e6ca with delete_type TABLET_DATA_TOMBSTONED (TS a04a37fc4aa7460cbfdbeeb4e451e1f5 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:36784
I20260925 03:04:07.797380 25355 tablet_replica.cc:330] T c5aec38e66a947c681e6c2a72b05e6ca P a04a37fc4aa7460cbfdbeeb4e451e1f5: stopping tablet replica
I20260925 03:04:07.799189 25369 ts_tablet_manager.cc:1918] T 71f461a6e2ff434993b78a2d077f0abd P 869b0a8575b54d08ae2817fad5dd42d5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260925 03:04:07.801213 25369 log.cc:1196] T 71f461a6e2ff434993b78a2d077f0abd P 869b0a8575b54d08ae2817fad5dd42d5: Deleting WAL directory at /tmp/dist-test-tasklAj8hX/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790305368086980-23300-0/minicluster-data/ts-0-root/wals/71f461a6e2ff434993b78a2d077f0abd
I20260925 03:04:07.799137 25355 raft_consensus.cc:2229] T c5aec38e66a947c681e6c2a72b05e6ca P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 03:04:07.805812 24542 consensus_queue.cc:237] T b4ebd2b043024a548005c8b6b00b0449 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:07.806365 24358 catalog_manager.cc:4841] TS 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187): tablet 71f461a6e2ff434993b78a2d077f0abd (table test-workload [id=f20271173d58474897a63e58ef5a856d]) successfully deleted
I20260925 03:04:07.803687 25355 raft_consensus.cc:2258] T c5aec38e66a947c681e6c2a72b05e6ca P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 03:04:07.814970 24467 raft_consensus.cc:1261] T b4ebd2b043024a548005c8b6b00b0449 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260925 03:04:07.816824 25355 ts_tablet_manager.cc:1905] T c5aec38e66a947c681e6c2a72b05e6ca P a04a37fc4aa7460cbfdbeeb4e451e1f5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260925 03:04:07.817248 25199 raft_consensus.cc:1261] T b4ebd2b043024a548005c8b6b00b0449 P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260925 03:04:07.817852 25353 consensus_queue.cc:1035] T b4ebd2b043024a548005c8b6b00b0449 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260925 03:04:07.819418 25370 consensus_queue.cc:1035] T b4ebd2b043024a548005c8b6b00b0449 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260925 03:04:07.829130 25306 raft_consensus.cc:2932] T b4ebd2b043024a548005c8b6b00b0449 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER eeab93abfdb141a895002ec9f312296c (127.22.193.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:07.830405 24467 raft_consensus.cc:2932] T b4ebd2b043024a548005c8b6b00b0449 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER eeab93abfdb141a895002ec9f312296c (127.22.193.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:07.843667 25355 ts_tablet_manager.cc:1918] T c5aec38e66a947c681e6c2a72b05e6ca P a04a37fc4aa7460cbfdbeeb4e451e1f5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260925 03:04:07.844117 25355 log.cc:1196] T c5aec38e66a947c681e6c2a72b05e6ca P a04a37fc4aa7460cbfdbeeb4e451e1f5: Deleting WAL directory at /tmp/dist-test-tasklAj8hX/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790305368086980-23300-0/minicluster-data/ts-1-root/wals/c5aec38e66a947c681e6c2a72b05e6ca
I20260925 03:04:07.846051 24356 catalog_manager.cc:4841] TS a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389): tablet c5aec38e66a947c681e6c2a72b05e6ca (table test-workload [id=f20271173d58474897a63e58ef5a856d]) successfully deleted
I20260925 03:04:07.847085 24356 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet b4ebd2b043024a548005c8b6b00b0449 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260925 03:04:07.842521 25199 raft_consensus.cc:2932] T b4ebd2b043024a548005c8b6b00b0449 P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER eeab93abfdb141a895002ec9f312296c (127.22.193.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:07.852037 24369 catalog_manager.cc:5495] T b4ebd2b043024a548005c8b6b00b0449 P 869b0a8575b54d08ae2817fad5dd42d5 reported cstate change: config changed from index 3 to 4, VOTER eeab93abfdb141a895002ec9f312296c (127.22.193.3) evicted. New cstate: current_term: 1 leader_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:07.864161 24606 tablet_service.cc:1514] Processing DeleteTablet for tablet b4ebd2b043024a548005c8b6b00b0449 with delete_type TABLET_DATA_TOMBSTONED (TS eeab93abfdb141a895002ec9f312296c not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:38550
I20260925 03:04:07.867245 25372 tablet_replica.cc:330] T b4ebd2b043024a548005c8b6b00b0449 P eeab93abfdb141a895002ec9f312296c: stopping tablet replica
I20260925 03:04:07.868232 25372 raft_consensus.cc:2229] T b4ebd2b043024a548005c8b6b00b0449 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 03:04:07.868870 25372 raft_consensus.cc:2258] T b4ebd2b043024a548005c8b6b00b0449 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 03:04:07.871856 25372 ts_tablet_manager.cc:1905] T b4ebd2b043024a548005c8b6b00b0449 P eeab93abfdb141a895002ec9f312296c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260925 03:04:07.895059 25372 ts_tablet_manager.cc:1918] T b4ebd2b043024a548005c8b6b00b0449 P eeab93abfdb141a895002ec9f312296c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260925 03:04:07.895664 25372 log.cc:1196] T b4ebd2b043024a548005c8b6b00b0449 P eeab93abfdb141a895002ec9f312296c: Deleting WAL directory at /tmp/dist-test-tasklAj8hX/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790305368086980-23300-0/minicluster-data/ts-2-root/wals/b4ebd2b043024a548005c8b6b00b0449
I20260925 03:04:07.898146 24355 catalog_manager.cc:4841] TS eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105): tablet b4ebd2b043024a548005c8b6b00b0449 (table test-workload [id=f20271173d58474897a63e58ef5a856d]) successfully deleted
I20260925 03:04:08.925815 24543 consensus_queue.cc:237] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } }
I20260925 03:04:08.932186 24470 raft_consensus.cc:1261] T e6b6ec010caf4689b8c90449764f5cf9 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 03:04:08.932436 24615 raft_consensus.cc:1261] T e6b6ec010caf4689b8c90449764f5cf9 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260925 03:04:08.933964 24511 consensus_peers.cc:487] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 -> Peer 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979): Couldn't send request to peer 1bc1f2e62ab44af995165bd5116a9d43. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e6b6ec010caf4689b8c90449764f5cf9. This is attempt 1: this message will repeat every 5th retry.
I20260925 03:04:08.934309 25353 consensus_queue.cc:1035] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260925 03:04:08.935307 25311 consensus_queue.cc:1035] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260925 03:04:08.945611 25375 raft_consensus.cc:2932] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } } }
I20260925 03:04:08.947461 24470 raft_consensus.cc:2932] T e6b6ec010caf4689b8c90449764f5cf9 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } } }
I20260925 03:04:08.948877 24615 raft_consensus.cc:2932] T e6b6ec010caf4689b8c90449764f5cf9 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } } }
I20260925 03:04:08.962711 24367 catalog_manager.cc:5495] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 reported cstate change: config changed from index -1 to 2, NON_VOTER 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) added. New cstate: current_term: 1 leader_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260925 03:04:08.970057 24470 consensus_queue.cc:237] T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } }
W20260925 03:04:08.976574 24436 consensus_peers.cc:487] T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5 -> Peer 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979): Couldn't send request to peer 1bc1f2e62ab44af995165bd5116a9d43. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b1e85248d3b04b979dd080caeeb02fdb. This is attempt 1: this message will repeat every 5th retry.
I20260925 03:04:08.978750 24615 raft_consensus.cc:1261] T b1e85248d3b04b979dd080caeeb02fdb P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 03:04:08.980403 24542 raft_consensus.cc:1261] T b1e85248d3b04b979dd080caeeb02fdb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 03:04:08.980258 25099 consensus_queue.cc:1035] T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260925 03:04:08.982764 25363 consensus_queue.cc:1035] T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260925 03:04:08.995401 25363 raft_consensus.cc:2932] T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } } }
I20260925 03:04:08.997076 24615 raft_consensus.cc:2932] T b1e85248d3b04b979dd080caeeb02fdb P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } } }
I20260925 03:04:08.997900 24542 raft_consensus.cc:2932] T b1e85248d3b04b979dd080caeeb02fdb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } } }
I20260925 03:04:09.011773 24369 catalog_manager.cc:5495] T b1e85248d3b04b979dd080caeeb02fdb P a04a37fc4aa7460cbfdbeeb4e451e1f5 reported cstate change: config changed from index -1 to 2, NON_VOTER 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) added. New cstate: current_term: 1 leader_uuid: "869b0a8575b54d08ae2817fad5dd42d5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } } }
I20260925 03:04:09.020035 24542 consensus_queue.cc:237] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } }
I20260925 03:04:09.081126 24467 raft_consensus.cc:1261] T 293ef69caf514ac98510a68495f54eda P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 03:04:09.083356 25375 consensus_queue.cc:1035] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260925 03:04:09.099560 24615 raft_consensus.cc:1261] T 293ef69caf514ac98510a68495f54eda P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 03:04:09.103622 25311 consensus_queue.cc:1035] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260925 03:04:09.102818 25390 raft_consensus.cc:2932] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } } }
W20260925 03:04:09.105664 24511 consensus_peers.cc:487] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 -> Peer 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979): Couldn't send request to peer 1bc1f2e62ab44af995165bd5116a9d43. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 293ef69caf514ac98510a68495f54eda. This is attempt 1: this message will repeat every 5th retry.
I20260925 03:04:09.130203 24622 raft_consensus.cc:2932] T 293ef69caf514ac98510a68495f54eda P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } } }
I20260925 03:04:09.176348 24542 consensus_queue.cc:237] T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } }
I20260925 03:04:09.182996 24369 catalog_manager.cc:5495] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 reported cstate change: config changed from index -1 to 2, NON_VOTER 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) added. New cstate: current_term: 1 leader_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260925 03:04:09.189622 24467 raft_consensus.cc:2932] T 293ef69caf514ac98510a68495f54eda P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } } }
W20260925 03:04:09.227370 24511 consensus_peers.cc:487] T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5 -> Peer 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979): Couldn't send request to peer 1bc1f2e62ab44af995165bd5116a9d43. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 04e068a0f80941dc9138cee063f53bab. This is attempt 1: this message will repeat every 5th retry.
I20260925 03:04:09.229200 24467 raft_consensus.cc:1261] T 04e068a0f80941dc9138cee063f53bab P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 03:04:09.249212 24622 raft_consensus.cc:1261] T 04e068a0f80941dc9138cee063f53bab P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 03:04:09.249594 25311 consensus_queue.cc:1035] T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260925 03:04:09.254504 25311 consensus_queue.cc:1035] T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260925 03:04:09.265156 25390 raft_consensus.cc:2932] T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } } }
I20260925 03:04:09.266757 24467 raft_consensus.cc:2932] T 04e068a0f80941dc9138cee063f53bab P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } } }
I20260925 03:04:09.267041 24622 raft_consensus.cc:2932] T 04e068a0f80941dc9138cee063f53bab P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } } }
I20260925 03:04:09.282960 24371 catalog_manager.cc:5495] T 04e068a0f80941dc9138cee063f53bab P 869b0a8575b54d08ae2817fad5dd42d5 reported cstate change: config changed from index -1 to 2, NON_VOTER 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) added. New cstate: current_term: 1 leader_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } } }
I20260925 03:04:09.485255 25399 ts_tablet_manager.cc:927] T b1e85248d3b04b979dd080caeeb02fdb P 1bc1f2e62ab44af995165bd5116a9d43: Initiating tablet copy from peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187)
I20260925 03:04:09.488489 25399 tablet_copy_client.cc:316] T b1e85248d3b04b979dd080caeeb02fdb P 1bc1f2e62ab44af995165bd5116a9d43: tablet copy: Beginning tablet copy session from remote peer at address 127.22.193.1:42187
I20260925 03:04:09.490832 24483 tablet_copy_service.cc:140] P 869b0a8575b54d08ae2817fad5dd42d5: Received BeginTabletCopySession request for tablet b1e85248d3b04b979dd080caeeb02fdb from peer 1bc1f2e62ab44af995165bd5116a9d43 ({username='slave'} at 127.0.0.1:38920)
I20260925 03:04:09.491438 24483 tablet_copy_service.cc:161] P 869b0a8575b54d08ae2817fad5dd42d5: Beginning new tablet copy session on tablet b1e85248d3b04b979dd080caeeb02fdb from peer 1bc1f2e62ab44af995165bd5116a9d43 at {username='slave'} at 127.0.0.1:38920: session id = 1bc1f2e62ab44af995165bd5116a9d43-b1e85248d3b04b979dd080caeeb02fdb
I20260925 03:04:09.500607 24483 tablet_copy_source_session.cc:205] T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 03:04:09.505321 25399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1e85248d3b04b979dd080caeeb02fdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:04:09.515785 25402 ts_tablet_manager.cc:927] T e6b6ec010caf4689b8c90449764f5cf9 P 1bc1f2e62ab44af995165bd5116a9d43: Initiating tablet copy from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:09.532603 25401 ts_tablet_manager.cc:927] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43: Initiating tablet copy from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:09.533601 25399 tablet_copy_client.cc:791] T b1e85248d3b04b979dd080caeeb02fdb P 1bc1f2e62ab44af995165bd5116a9d43: tablet copy: Starting download of 0 data blocks...
I20260925 03:04:09.534888 25399 tablet_copy_client.cc:655] T b1e85248d3b04b979dd080caeeb02fdb P 1bc1f2e62ab44af995165bd5116a9d43: tablet copy: Starting download of 1 WAL segments...
I20260925 03:04:09.536286 25402 tablet_copy_client.cc:316] T e6b6ec010caf4689b8c90449764f5cf9 P 1bc1f2e62ab44af995165bd5116a9d43: tablet copy: Beginning tablet copy session from remote peer at address 127.22.193.2:36389
I20260925 03:04:09.539690 25399 tablet_copy_client.cc:531] T b1e85248d3b04b979dd080caeeb02fdb P 1bc1f2e62ab44af995165bd5116a9d43: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 03:04:09.541060 24558 tablet_copy_service.cc:140] P a04a37fc4aa7460cbfdbeeb4e451e1f5: Received BeginTabletCopySession request for tablet e6b6ec010caf4689b8c90449764f5cf9 from peer 1bc1f2e62ab44af995165bd5116a9d43 ({username='slave'} at 127.0.0.1:33068)
I20260925 03:04:09.541615 24558 tablet_copy_service.cc:161] P a04a37fc4aa7460cbfdbeeb4e451e1f5: Beginning new tablet copy session on tablet e6b6ec010caf4689b8c90449764f5cf9 from peer 1bc1f2e62ab44af995165bd5116a9d43 at {username='slave'} at 127.0.0.1:33068: session id = 1bc1f2e62ab44af995165bd5116a9d43-e6b6ec010caf4689b8c90449764f5cf9
I20260925 03:04:09.549947 25399 tablet_bootstrap.cc:492] T b1e85248d3b04b979dd080caeeb02fdb P 1bc1f2e62ab44af995165bd5116a9d43: Bootstrap starting.
I20260925 03:04:09.551544 24558 tablet_copy_source_session.cc:205] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 03:04:09.552927 25401 tablet_copy_client.cc:316] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43: tablet copy: Beginning tablet copy session from remote peer at address 127.22.193.2:36389
I20260925 03:04:09.555886 24558 tablet_copy_service.cc:140] P a04a37fc4aa7460cbfdbeeb4e451e1f5: Received BeginTabletCopySession request for tablet 293ef69caf514ac98510a68495f54eda from peer 1bc1f2e62ab44af995165bd5116a9d43 ({username='slave'} at 127.0.0.1:33068)
I20260925 03:04:09.556166 25402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6b6ec010caf4689b8c90449764f5cf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:04:09.556787 24558 tablet_copy_service.cc:161] P a04a37fc4aa7460cbfdbeeb4e451e1f5: Beginning new tablet copy session on tablet 293ef69caf514ac98510a68495f54eda from peer 1bc1f2e62ab44af995165bd5116a9d43 at {username='slave'} at 127.0.0.1:33068: session id = 1bc1f2e62ab44af995165bd5116a9d43-293ef69caf514ac98510a68495f54eda
I20260925 03:04:09.571121 24558 tablet_copy_source_session.cc:205] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 03:04:09.575294 25401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 293ef69caf514ac98510a68495f54eda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:04:09.578280 25402 tablet_copy_client.cc:791] T e6b6ec010caf4689b8c90449764f5cf9 P 1bc1f2e62ab44af995165bd5116a9d43: tablet copy: Starting download of 0 data blocks...
I20260925 03:04:09.578992 25402 tablet_copy_client.cc:655] T e6b6ec010caf4689b8c90449764f5cf9 P 1bc1f2e62ab44af995165bd5116a9d43: tablet copy: Starting download of 1 WAL segments...
I20260925 03:04:09.583706 25402 tablet_copy_client.cc:531] T e6b6ec010caf4689b8c90449764f5cf9 P 1bc1f2e62ab44af995165bd5116a9d43: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 03:04:09.585325 25399 tablet_bootstrap.cc:492] T b1e85248d3b04b979dd080caeeb02fdb P 1bc1f2e62ab44af995165bd5116a9d43: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 03:04:09.586324 25399 tablet_bootstrap.cc:492] T b1e85248d3b04b979dd080caeeb02fdb P 1bc1f2e62ab44af995165bd5116a9d43: Bootstrap complete.
I20260925 03:04:09.587051 25399 ts_tablet_manager.cc:1397] T b1e85248d3b04b979dd080caeeb02fdb P 1bc1f2e62ab44af995165bd5116a9d43: Time spent bootstrapping tablet: real 0.037s user 0.016s sys 0.011s
I20260925 03:04:09.590165 25399 raft_consensus.cc:348] T b1e85248d3b04b979dd080caeeb02fdb P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } }
I20260925 03:04:09.591162 25399 raft_consensus.cc:729] T b1e85248d3b04b979dd080caeeb02fdb P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1bc1f2e62ab44af995165bd5116a9d43, State: Initialized, Role: LEARNER
I20260925 03:04:09.591902 25399 consensus_queue.cc:260] T b1e85248d3b04b979dd080caeeb02fdb P 1bc1f2e62ab44af995165bd5116a9d43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } }
I20260925 03:04:09.598379 25402 tablet_bootstrap.cc:492] T e6b6ec010caf4689b8c90449764f5cf9 P 1bc1f2e62ab44af995165bd5116a9d43: Bootstrap starting.
I20260925 03:04:09.599784 25401 tablet_copy_client.cc:791] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43: tablet copy: Starting download of 0 data blocks...
I20260925 03:04:09.601004 25401 tablet_copy_client.cc:655] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43: tablet copy: Starting download of 1 WAL segments...
I20260925 03:04:09.604751 25401 tablet_copy_client.cc:531] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 03:04:09.607074 25399 ts_tablet_manager.cc:1428] T b1e85248d3b04b979dd080caeeb02fdb P 1bc1f2e62ab44af995165bd5116a9d43: Time spent starting tablet: real 0.020s user 0.007s sys 0.007s
I20260925 03:04:09.609025 24483 tablet_copy_service.cc:342] P 869b0a8575b54d08ae2817fad5dd42d5: Request end of tablet copy session 1bc1f2e62ab44af995165bd5116a9d43-b1e85248d3b04b979dd080caeeb02fdb received from {username='slave'} at 127.0.0.1:38920
I20260925 03:04:09.609452 24483 tablet_copy_service.cc:433] P 869b0a8575b54d08ae2817fad5dd42d5: ending tablet copy session 1bc1f2e62ab44af995165bd5116a9d43-b1e85248d3b04b979dd080caeeb02fdb on tablet b1e85248d3b04b979dd080caeeb02fdb with peer 1bc1f2e62ab44af995165bd5116a9d43
I20260925 03:04:09.617700 25401 tablet_bootstrap.cc:492] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43: Bootstrap starting.
I20260925 03:04:09.636268 25402 tablet_bootstrap.cc:492] T e6b6ec010caf4689b8c90449764f5cf9 P 1bc1f2e62ab44af995165bd5116a9d43: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 03:04:09.637125 25402 tablet_bootstrap.cc:492] T e6b6ec010caf4689b8c90449764f5cf9 P 1bc1f2e62ab44af995165bd5116a9d43: Bootstrap complete.
I20260925 03:04:09.637820 25402 ts_tablet_manager.cc:1397] T e6b6ec010caf4689b8c90449764f5cf9 P 1bc1f2e62ab44af995165bd5116a9d43: Time spent bootstrapping tablet: real 0.040s user 0.023s sys 0.000s
I20260925 03:04:09.639540 25401 tablet_bootstrap.cc:492] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 03:04:09.640460 25401 tablet_bootstrap.cc:492] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43: Bootstrap complete.
I20260925 03:04:09.641181 25401 ts_tablet_manager.cc:1397] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43: Time spent bootstrapping tablet: real 0.024s user 0.013s sys 0.010s
I20260925 03:04:09.641803 25402 raft_consensus.cc:348] T e6b6ec010caf4689b8c90449764f5cf9 P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } }
I20260925 03:04:09.642968 25402 raft_consensus.cc:729] T e6b6ec010caf4689b8c90449764f5cf9 P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1bc1f2e62ab44af995165bd5116a9d43, State: Initialized, Role: LEARNER
I20260925 03:04:09.643626 25402 consensus_queue.cc:260] T e6b6ec010caf4689b8c90449764f5cf9 P 1bc1f2e62ab44af995165bd5116a9d43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } }
I20260925 03:04:09.643815 25401 raft_consensus.cc:348] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } }
I20260925 03:04:09.644652 25401 raft_consensus.cc:729] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1bc1f2e62ab44af995165bd5116a9d43, State: Initialized, Role: LEARNER
I20260925 03:04:09.645344 25401 consensus_queue.cc:260] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } }
I20260925 03:04:09.645903 25402 ts_tablet_manager.cc:1428] T e6b6ec010caf4689b8c90449764f5cf9 P 1bc1f2e62ab44af995165bd5116a9d43: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260925 03:04:09.647825 25401 ts_tablet_manager.cc:1428] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260925 03:04:09.647746 24558 tablet_copy_service.cc:342] P a04a37fc4aa7460cbfdbeeb4e451e1f5: Request end of tablet copy session 1bc1f2e62ab44af995165bd5116a9d43-e6b6ec010caf4689b8c90449764f5cf9 received from {username='slave'} at 127.0.0.1:33068
I20260925 03:04:09.648366 24558 tablet_copy_service.cc:433] P a04a37fc4aa7460cbfdbeeb4e451e1f5: ending tablet copy session 1bc1f2e62ab44af995165bd5116a9d43-e6b6ec010caf4689b8c90449764f5cf9 on tablet e6b6ec010caf4689b8c90449764f5cf9 with peer 1bc1f2e62ab44af995165bd5116a9d43
I20260925 03:04:09.657620 24558 tablet_copy_service.cc:342] P a04a37fc4aa7460cbfdbeeb4e451e1f5: Request end of tablet copy session 1bc1f2e62ab44af995165bd5116a9d43-293ef69caf514ac98510a68495f54eda received from {username='slave'} at 127.0.0.1:33068
I20260925 03:04:09.658016 24558 tablet_copy_service.cc:433] P a04a37fc4aa7460cbfdbeeb4e451e1f5: ending tablet copy session 1bc1f2e62ab44af995165bd5116a9d43-293ef69caf514ac98510a68495f54eda on tablet 293ef69caf514ac98510a68495f54eda with peer 1bc1f2e62ab44af995165bd5116a9d43
I20260925 03:04:09.820905 25402 ts_tablet_manager.cc:927] T 04e068a0f80941dc9138cee063f53bab P 1bc1f2e62ab44af995165bd5116a9d43: Initiating tablet copy from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:09.822520 25402 tablet_copy_client.cc:316] T 04e068a0f80941dc9138cee063f53bab P 1bc1f2e62ab44af995165bd5116a9d43: tablet copy: Beginning tablet copy session from remote peer at address 127.22.193.2:36389
I20260925 03:04:09.823792 24558 tablet_copy_service.cc:140] P a04a37fc4aa7460cbfdbeeb4e451e1f5: Received BeginTabletCopySession request for tablet 04e068a0f80941dc9138cee063f53bab from peer 1bc1f2e62ab44af995165bd5116a9d43 ({username='slave'} at 127.0.0.1:33068)
I20260925 03:04:09.824312 24558 tablet_copy_service.cc:161] P a04a37fc4aa7460cbfdbeeb4e451e1f5: Beginning new tablet copy session on tablet 04e068a0f80941dc9138cee063f53bab from peer 1bc1f2e62ab44af995165bd5116a9d43 at {username='slave'} at 127.0.0.1:33068: session id = 1bc1f2e62ab44af995165bd5116a9d43-04e068a0f80941dc9138cee063f53bab
I20260925 03:04:09.832258 24558 tablet_copy_source_session.cc:205] T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 03:04:09.835472 25402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04e068a0f80941dc9138cee063f53bab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:04:09.852098 25402 tablet_copy_client.cc:791] T 04e068a0f80941dc9138cee063f53bab P 1bc1f2e62ab44af995165bd5116a9d43: tablet copy: Starting download of 0 data blocks...
I20260925 03:04:09.852667 25402 tablet_copy_client.cc:655] T 04e068a0f80941dc9138cee063f53bab P 1bc1f2e62ab44af995165bd5116a9d43: tablet copy: Starting download of 1 WAL segments...
I20260925 03:04:09.855931 25402 tablet_copy_client.cc:531] T 04e068a0f80941dc9138cee063f53bab P 1bc1f2e62ab44af995165bd5116a9d43: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 03:04:09.864290 25402 tablet_bootstrap.cc:492] T 04e068a0f80941dc9138cee063f53bab P 1bc1f2e62ab44af995165bd5116a9d43: Bootstrap starting.
I20260925 03:04:09.883143 25402 tablet_bootstrap.cc:492] T 04e068a0f80941dc9138cee063f53bab P 1bc1f2e62ab44af995165bd5116a9d43: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 03:04:09.883929 25402 tablet_bootstrap.cc:492] T 04e068a0f80941dc9138cee063f53bab P 1bc1f2e62ab44af995165bd5116a9d43: Bootstrap complete.
I20260925 03:04:09.884563 25402 ts_tablet_manager.cc:1397] T 04e068a0f80941dc9138cee063f53bab P 1bc1f2e62ab44af995165bd5116a9d43: Time spent bootstrapping tablet: real 0.020s user 0.018s sys 0.000s
I20260925 03:04:09.887001 25402 raft_consensus.cc:348] T 04e068a0f80941dc9138cee063f53bab P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } }
I20260925 03:04:09.887817 25402 raft_consensus.cc:729] T 04e068a0f80941dc9138cee063f53bab P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1bc1f2e62ab44af995165bd5116a9d43, State: Initialized, Role: LEARNER
I20260925 03:04:09.888332 25402 consensus_queue.cc:260] T 04e068a0f80941dc9138cee063f53bab P 1bc1f2e62ab44af995165bd5116a9d43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } }
I20260925 03:04:09.893774 25402 ts_tablet_manager.cc:1428] T 04e068a0f80941dc9138cee063f53bab P 1bc1f2e62ab44af995165bd5116a9d43: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20260925 03:04:09.895404 24558 tablet_copy_service.cc:342] P a04a37fc4aa7460cbfdbeeb4e451e1f5: Request end of tablet copy session 1bc1f2e62ab44af995165bd5116a9d43-04e068a0f80941dc9138cee063f53bab received from {username='slave'} at 127.0.0.1:33068
I20260925 03:04:09.895795 24558 tablet_copy_service.cc:433] P a04a37fc4aa7460cbfdbeeb4e451e1f5: ending tablet copy session 1bc1f2e62ab44af995165bd5116a9d43-04e068a0f80941dc9138cee063f53bab on tablet 04e068a0f80941dc9138cee063f53bab with peer 1bc1f2e62ab44af995165bd5116a9d43
I20260925 03:04:09.933518 25197 raft_consensus.cc:1203] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260925 03:04:09.943416 25199 raft_consensus.cc:1203] T e6b6ec010caf4689b8c90449764f5cf9 P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260925 03:04:09.975751 25197 raft_consensus.cc:1203] T b1e85248d3b04b979dd080caeeb02fdb P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260925 03:04:10.384524 25390 raft_consensus.cc:1050] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5: attempting to promote NON_VOTER 1bc1f2e62ab44af995165bd5116a9d43 to VOTER
I20260925 03:04:10.388939 25390 consensus_queue.cc:237] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:10.395335 25197 raft_consensus.cc:1261] T e6b6ec010caf4689b8c90449764f5cf9 P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 LEARNER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260925 03:04:10.396840 24622 raft_consensus.cc:1261] T e6b6ec010caf4689b8c90449764f5cf9 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260925 03:04:10.397557 25390 consensus_queue.cc:1035] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260925 03:04:10.399184 24467 raft_consensus.cc:1261] T e6b6ec010caf4689b8c90449764f5cf9 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260925 03:04:10.400890 25390 consensus_queue.cc:1035] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260925 03:04:10.405185 25390 consensus_queue.cc:1035] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260925 03:04:10.424453 25375 raft_consensus.cc:2932] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:10.426193 25197 raft_consensus.cc:2932] T e6b6ec010caf4689b8c90449764f5cf9 P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:10.428891 24622 raft_consensus.cc:2932] T e6b6ec010caf4689b8c90449764f5cf9 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:10.431445 25199 raft_consensus.cc:1203] T 04e068a0f80941dc9138cee063f53bab P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260925 03:04:10.438126 25385 raft_consensus.cc:1050] T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5: attempting to promote NON_VOTER 1bc1f2e62ab44af995165bd5116a9d43 to VOTER
I20260925 03:04:10.441110 25385 consensus_queue.cc:237] T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:10.444378 24467 raft_consensus.cc:2932] T e6b6ec010caf4689b8c90449764f5cf9 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:10.452064 24369 catalog_manager.cc:5495] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 reported cstate change: config changed from index 2 to 3, 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260925 03:04:10.460727 25394 raft_consensus.cc:1050] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5: attempting to promote NON_VOTER 1bc1f2e62ab44af995165bd5116a9d43 to VOTER
I20260925 03:04:10.469238 24615 raft_consensus.cc:1261] T b1e85248d3b04b979dd080caeeb02fdb P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260925 03:04:10.471053 25385 consensus_queue.cc:1035] T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20260925 03:04:10.476006 24543 raft_consensus.cc:1261] T b1e85248d3b04b979dd080caeeb02fdb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260925 03:04:10.476591 25198 raft_consensus.cc:1261] T b1e85248d3b04b979dd080caeeb02fdb P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 LEARNER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260925 03:04:10.478427 25385 consensus_queue.cc:1035] T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260925 03:04:10.479979 25391 consensus_queue.cc:1035] T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260925 03:04:10.500064 25394 consensus_queue.cc:237] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:10.547479 25389 raft_consensus.cc:2932] T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:10.553660 24467 raft_consensus.cc:1261] T 293ef69caf514ac98510a68495f54eda P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260925 03:04:10.555297 24615 raft_consensus.cc:1261] T 293ef69caf514ac98510a68495f54eda P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260925 03:04:10.556041 25198 raft_consensus.cc:1261] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 LEARNER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260925 03:04:10.556732 25390 consensus_queue.cc:1035] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260925 03:04:10.555058 24543 raft_consensus.cc:2932] T b1e85248d3b04b979dd080caeeb02fdb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:10.559055 25375 consensus_queue.cc:1035] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260925 03:04:10.559862 25390 consensus_queue.cc:1035] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260925 03:04:10.583767 25197 raft_consensus.cc:2932] T b1e85248d3b04b979dd080caeeb02fdb P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:10.584589 24622 raft_consensus.cc:2932] T b1e85248d3b04b979dd080caeeb02fdb P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:10.592916 24467 raft_consensus.cc:2932] T 293ef69caf514ac98510a68495f54eda P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:10.599063 24542 consensus_queue.cc:237] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:10.593818 25394 raft_consensus.cc:2932] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:10.607380 24367 catalog_manager.cc:5495] T b1e85248d3b04b979dd080caeeb02fdb P a04a37fc4aa7460cbfdbeeb4e451e1f5 reported cstate change: config changed from index 2 to 3, 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "869b0a8575b54d08ae2817fad5dd42d5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:10.622924 25199 raft_consensus.cc:1261] T e6b6ec010caf4689b8c90449764f5cf9 P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260925 03:04:10.626820 25418 consensus_queue.cc:1035] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260925 03:04:10.602584 25198 raft_consensus.cc:2932] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:10.634182 24470 raft_consensus.cc:1261] T e6b6ec010caf4689b8c90449764f5cf9 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260925 03:04:10.633137 24467 consensus_queue.cc:237] T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:10.637526 25390 consensus_queue.cc:1035] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260925 03:04:10.639405 25390 raft_consensus.cc:2932] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER eeab93abfdb141a895002ec9f312296c (127.22.193.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:10.626621 24622 raft_consensus.cc:2932] T 293ef69caf514ac98510a68495f54eda P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:10.644320 24464 raft_consensus.cc:2932] T e6b6ec010caf4689b8c90449764f5cf9 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER eeab93abfdb141a895002ec9f312296c (127.22.193.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:10.650205 25199 raft_consensus.cc:2932] T e6b6ec010caf4689b8c90449764f5cf9 P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER eeab93abfdb141a895002ec9f312296c (127.22.193.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:10.652302 24614 raft_consensus.cc:1261] T b1e85248d3b04b979dd080caeeb02fdb P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260925 03:04:10.656788 25385 consensus_queue.cc:1035] T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260925 03:04:10.652522 25198 raft_consensus.cc:1261] T b1e85248d3b04b979dd080caeeb02fdb P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260925 03:04:10.662637 25385 consensus_queue.cc:1035] T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260925 03:04:10.664140 24371 catalog_manager.cc:5495] T 293ef69caf514ac98510a68495f54eda P 869b0a8575b54d08ae2817fad5dd42d5 reported cstate change: config changed from index 2 to 3, 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:10.671002 25389 raft_consensus.cc:2932] T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:10.683477 25199 raft_consensus.cc:2932] T b1e85248d3b04b979dd080caeeb02fdb P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:10.689433 24371 catalog_manager.cc:5495] T e6b6ec010caf4689b8c90449764f5cf9 P 1bc1f2e62ab44af995165bd5116a9d43 reported cstate change: config changed from index 3 to 4, VOTER eeab93abfdb141a895002ec9f312296c (127.22.193.3) evicted. New cstate: current_term: 1 leader_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:10.691769 24614 raft_consensus.cc:2932] T b1e85248d3b04b979dd080caeeb02fdb P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:10.694599 24543 consensus_queue.cc:237] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:10.698136 24356 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet e6b6ec010caf4689b8c90449764f5cf9 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260925 03:04:10.701777 25199 raft_consensus.cc:1261] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260925 03:04:10.703068 25394 consensus_queue.cc:1035] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260925 03:04:10.707407 24464 raft_consensus.cc:1261] T 293ef69caf514ac98510a68495f54eda P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260925 03:04:10.709731 24358 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet b1e85248d3b04b979dd080caeeb02fdb with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260925 03:04:10.715373 24606 tablet_service.cc:1514] Processing DeleteTablet for tablet e6b6ec010caf4689b8c90449764f5cf9 with delete_type TABLET_DATA_TOMBSTONED (TS eeab93abfdb141a895002ec9f312296c not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:38550
I20260925 03:04:10.724470 25418 consensus_queue.cc:1035] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260925 03:04:10.724764 24367 catalog_manager.cc:5495] T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5 reported cstate change: config changed from index 3 to 4, VOTER a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2) evicted. New cstate: current_term: 1 leader_uuid: "869b0a8575b54d08ae2817fad5dd42d5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260925 03:04:10.733177 25375 raft_consensus.cc:2932] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER eeab93abfdb141a895002ec9f312296c (127.22.193.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:10.734408 24467 raft_consensus.cc:2932] T 293ef69caf514ac98510a68495f54eda P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER eeab93abfdb141a895002ec9f312296c (127.22.193.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:10.742808 24534 tablet_service.cc:1514] Processing DeleteTablet for tablet b1e85248d3b04b979dd080caeeb02fdb with delete_type TABLET_DATA_TOMBSTONED (TS a04a37fc4aa7460cbfdbeeb4e451e1f5 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:36784
I20260925 03:04:10.745438 25199 raft_consensus.cc:2932] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER eeab93abfdb141a895002ec9f312296c (127.22.193.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:10.754750 24356 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 293ef69caf514ac98510a68495f54eda with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260925 03:04:10.763542 25434 tablet_replica.cc:330] T e6b6ec010caf4689b8c90449764f5cf9 P eeab93abfdb141a895002ec9f312296c: stopping tablet replica
I20260925 03:04:10.767820 25434 raft_consensus.cc:2229] T e6b6ec010caf4689b8c90449764f5cf9 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 03:04:10.772001 25434 raft_consensus.cc:2258] T e6b6ec010caf4689b8c90449764f5cf9 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 03:04:10.770426 25435 tablet_replica.cc:330] T b1e85248d3b04b979dd080caeeb02fdb P a04a37fc4aa7460cbfdbeeb4e451e1f5: stopping tablet replica
I20260925 03:04:10.773373 24368 catalog_manager.cc:5495] T 293ef69caf514ac98510a68495f54eda P 869b0a8575b54d08ae2817fad5dd42d5 reported cstate change: config changed from index 3 to 4, VOTER eeab93abfdb141a895002ec9f312296c (127.22.193.3) evicted. New cstate: current_term: 1 leader_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:10.775009 25435 raft_consensus.cc:2229] T b1e85248d3b04b979dd080caeeb02fdb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 03:04:10.775810 25435 raft_consensus.cc:2258] T b1e85248d3b04b979dd080caeeb02fdb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 03:04:10.776715 25434 ts_tablet_manager.cc:1905] T e6b6ec010caf4689b8c90449764f5cf9 P eeab93abfdb141a895002ec9f312296c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260925 03:04:10.781476 25435 ts_tablet_manager.cc:1905] T b1e85248d3b04b979dd080caeeb02fdb P a04a37fc4aa7460cbfdbeeb4e451e1f5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260925 03:04:10.800936 25434 ts_tablet_manager.cc:1918] T e6b6ec010caf4689b8c90449764f5cf9 P eeab93abfdb141a895002ec9f312296c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260925 03:04:10.801553 25434 log.cc:1196] T e6b6ec010caf4689b8c90449764f5cf9 P eeab93abfdb141a895002ec9f312296c: Deleting WAL directory at /tmp/dist-test-tasklAj8hX/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790305368086980-23300-0/minicluster-data/ts-2-root/wals/e6b6ec010caf4689b8c90449764f5cf9
I20260925 03:04:10.803781 24355 catalog_manager.cc:4841] TS eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105): tablet e6b6ec010caf4689b8c90449764f5cf9 (table test-workload [id=f20271173d58474897a63e58ef5a856d]) successfully deleted
I20260925 03:04:10.807060 25435 ts_tablet_manager.cc:1918] T b1e85248d3b04b979dd080caeeb02fdb P a04a37fc4aa7460cbfdbeeb4e451e1f5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260925 03:04:10.807592 25435 log.cc:1196] T b1e85248d3b04b979dd080caeeb02fdb P a04a37fc4aa7460cbfdbeeb4e451e1f5: Deleting WAL directory at /tmp/dist-test-tasklAj8hX/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790305368086980-23300-0/minicluster-data/ts-1-root/wals/b1e85248d3b04b979dd080caeeb02fdb
I20260925 03:04:10.810330 24606 tablet_service.cc:1514] Processing DeleteTablet for tablet 293ef69caf514ac98510a68495f54eda with delete_type TABLET_DATA_TOMBSTONED (TS eeab93abfdb141a895002ec9f312296c not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:38550
I20260925 03:04:10.811259 25434 tablet_replica.cc:330] T 293ef69caf514ac98510a68495f54eda P eeab93abfdb141a895002ec9f312296c: stopping tablet replica
I20260925 03:04:10.819824 24356 catalog_manager.cc:4841] TS a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389): tablet b1e85248d3b04b979dd080caeeb02fdb (table test-workload [id=f20271173d58474897a63e58ef5a856d]) successfully deleted
I20260925 03:04:10.826344 25434 raft_consensus.cc:2229] T 293ef69caf514ac98510a68495f54eda P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 03:04:10.827170 25434 raft_consensus.cc:2258] T 293ef69caf514ac98510a68495f54eda P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 03:04:10.831312 25434 ts_tablet_manager.cc:1905] T 293ef69caf514ac98510a68495f54eda P eeab93abfdb141a895002ec9f312296c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260925 03:04:10.846485 25434 ts_tablet_manager.cc:1918] T 293ef69caf514ac98510a68495f54eda P eeab93abfdb141a895002ec9f312296c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260925 03:04:10.847005 25434 log.cc:1196] T 293ef69caf514ac98510a68495f54eda P eeab93abfdb141a895002ec9f312296c: Deleting WAL directory at /tmp/dist-test-tasklAj8hX/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790305368086980-23300-0/minicluster-data/ts-2-root/wals/293ef69caf514ac98510a68495f54eda
I20260925 03:04:10.848300 24355 catalog_manager.cc:4841] TS eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105): tablet 293ef69caf514ac98510a68495f54eda (table test-workload [id=f20271173d58474897a63e58ef5a856d]) successfully deleted
I20260925 03:04:10.892246 25433 raft_consensus.cc:1050] T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5: attempting to promote NON_VOTER 1bc1f2e62ab44af995165bd5116a9d43 to VOTER
I20260925 03:04:10.897441 25433 consensus_queue.cc:237] T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:10.902932 25199 raft_consensus.cc:1261] T 04e068a0f80941dc9138cee063f53bab P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 LEARNER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260925 03:04:10.903074 24464 raft_consensus.cc:1261] T 04e068a0f80941dc9138cee063f53bab P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260925 03:04:10.904402 25375 consensus_queue.cc:1035] T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260925 03:04:10.905335 24622 raft_consensus.cc:1261] T 04e068a0f80941dc9138cee063f53bab P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260925 03:04:10.906054 25390 consensus_queue.cc:1035] T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260925 03:04:10.908438 25390 consensus_queue.cc:1035] T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260925 03:04:10.917884 25390 raft_consensus.cc:2932] T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:10.919355 24622 raft_consensus.cc:2932] T 04e068a0f80941dc9138cee063f53bab P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:10.930044 24464 raft_consensus.cc:2932] T 04e068a0f80941dc9138cee063f53bab P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:10.937016 24368 catalog_manager.cc:5495] T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5 reported cstate change: config changed from index 2 to 3, 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260925 03:04:10.942723 25199 raft_consensus.cc:2932] T 04e068a0f80941dc9138cee063f53bab P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:10.969182 24545 consensus_queue.cc:237] T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:10.975235 24614 raft_consensus.cc:1261] T 04e068a0f80941dc9138cee063f53bab P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260925 03:04:10.976924 25199 raft_consensus.cc:1261] T 04e068a0f80941dc9138cee063f53bab P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260925 03:04:10.979905 25390 consensus_queue.cc:1035] T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260925 03:04:10.980890 25433 consensus_queue.cc:1035] T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260925 03:04:10.989300 25433 raft_consensus.cc:2932] T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:10.995043 25199 raft_consensus.cc:2932] T 04e068a0f80941dc9138cee063f53bab P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:10.999295 24356 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 04e068a0f80941dc9138cee063f53bab with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260925 03:04:10.999142 24622 raft_consensus.cc:2932] T 04e068a0f80941dc9138cee063f53bab P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:11.011050 24368 catalog_manager.cc:5495] T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5 reported cstate change: config changed from index 3 to 4, VOTER 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1) evicted. New cstate: current_term: 1 leader_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260925 03:04:11.024152 24456 tablet_service.cc:1514] Processing DeleteTablet for tablet 04e068a0f80941dc9138cee063f53bab with delete_type TABLET_DATA_TOMBSTONED (TS 869b0a8575b54d08ae2817fad5dd42d5 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:55552
I20260925 03:04:11.026819 25440 tablet_replica.cc:330] T 04e068a0f80941dc9138cee063f53bab P 869b0a8575b54d08ae2817fad5dd42d5: stopping tablet replica
I20260925 03:04:11.027828 25440 raft_consensus.cc:2229] T 04e068a0f80941dc9138cee063f53bab P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 03:04:11.028507 25440 raft_consensus.cc:2258] T 04e068a0f80941dc9138cee063f53bab P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 03:04:11.031221 25440 ts_tablet_manager.cc:1905] T 04e068a0f80941dc9138cee063f53bab P 869b0a8575b54d08ae2817fad5dd42d5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260925 03:04:11.043963 25440 ts_tablet_manager.cc:1918] T 04e068a0f80941dc9138cee063f53bab P 869b0a8575b54d08ae2817fad5dd42d5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260925 03:04:11.044270 25440 log.cc:1196] T 04e068a0f80941dc9138cee063f53bab P 869b0a8575b54d08ae2817fad5dd42d5: Deleting WAL directory at /tmp/dist-test-tasklAj8hX/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790305368086980-23300-0/minicluster-data/ts-0-root/wals/04e068a0f80941dc9138cee063f53bab
I20260925 03:04:11.045512 24358 catalog_manager.cc:4841] TS 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187): tablet 04e068a0f80941dc9138cee063f53bab (table test-workload [id=f20271173d58474897a63e58ef5a856d]) successfully deleted
I20260925 03:04:12.065258 24464 consensus_queue.cc:237] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } }
I20260925 03:04:12.071013 24543 raft_consensus.cc:1261] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 03:04:12.071301 24615 raft_consensus.cc:1261] T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260925 03:04:12.071576 24436 consensus_peers.cc:487] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 -> Peer 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979): Couldn't send request to peer 1bc1f2e62ab44af995165bd5116a9d43. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7e49248643df4131a1c42e74244e05c2. This is attempt 1: this message will repeat every 5th retry.
I20260925 03:04:12.072329 25389 consensus_queue.cc:1035] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260925 03:04:12.073251 25412 consensus_queue.cc:1035] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260925 03:04:12.081866 25442 raft_consensus.cc:2932] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } } }
I20260925 03:04:12.086632 24615 raft_consensus.cc:2932] T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } } }
I20260925 03:04:12.100710 24371 catalog_manager.cc:5495] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 reported cstate change: config changed from index -1 to 2, NON_VOTER 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) added. New cstate: current_term: 1 leader_uuid: "869b0a8575b54d08ae2817fad5dd42d5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260925 03:04:12.107575 24622 consensus_queue.cc:237] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } }
I20260925 03:04:12.096912 24543 raft_consensus.cc:2932] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } } }
W20260925 03:04:12.160204 24585 consensus_peers.cc:487] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c -> Peer 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979): Couldn't send request to peer 1bc1f2e62ab44af995165bd5116a9d43. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 29637f5a4a504ce6ba29033ccd0cd716. This is attempt 1: this message will repeat every 5th retry.
I20260925 03:04:12.161772 24545 raft_consensus.cc:1261] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 03:04:12.166504 25422 consensus_queue.cc:1035] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260925 03:04:12.181213 24470 raft_consensus.cc:1261] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 03:04:12.183880 25422 consensus_queue.cc:1035] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260925 03:04:12.200624 25455 raft_consensus.cc:2932] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } } }
I20260925 03:04:12.205935 24545 raft_consensus.cc:2932] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } } }
I20260925 03:04:12.213989 24470 raft_consensus.cc:2932] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } } }
I20260925 03:04:12.231496 24622 consensus_queue.cc:237] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } attrs { replace: true } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } }
I20260925 03:04:12.235457 24369 catalog_manager.cc:5495] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 reported cstate change: config changed from index -1 to 2, NON_VOTER 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) added. New cstate: current_term: 1 leader_uuid: "eeab93abfdb141a895002ec9f312296c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } } }
I20260925 03:04:12.241833 24470 raft_consensus.cc:1261] T e92f377a79354a6ebb4eec67303c13e3 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260925 03:04:12.242066 24585 consensus_peers.cc:487] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c -> Peer 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979): Couldn't send request to peer 1bc1f2e62ab44af995165bd5116a9d43. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e92f377a79354a6ebb4eec67303c13e3. This is attempt 1: this message will repeat every 5th retry.
I20260925 03:04:12.242882 24545 raft_consensus.cc:1261] T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 03:04:12.243103 25422 consensus_queue.cc:1035] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260925 03:04:12.244169 25455 consensus_queue.cc:1035] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260925 03:04:12.255789 25454 raft_consensus.cc:2932] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } attrs { replace: true } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } } }
I20260925 03:04:12.257259 24470 raft_consensus.cc:2932] T e92f377a79354a6ebb4eec67303c13e3 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } attrs { replace: true } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } } }
I20260925 03:04:12.265249 24545 raft_consensus.cc:2932] T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } attrs { replace: true } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } } }
I20260925 03:04:12.276880 24371 catalog_manager.cc:5495] T e92f377a79354a6ebb4eec67303c13e3 P 869b0a8575b54d08ae2817fad5dd42d5 reported cstate change: config changed from index -1 to 2, NON_VOTER 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) added. New cstate: current_term: 1 leader_uuid: "eeab93abfdb141a895002ec9f312296c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } attrs { replace: true } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } } }
I20260925 03:04:12.280169 24470 consensus_queue.cc:237] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } }
I20260925 03:04:12.288673 24545 raft_consensus.cc:1261] T 367277daf8d04778b3fa4ed94d71580d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 03:04:12.290097 24622 raft_consensus.cc:1261] T 367277daf8d04778b3fa4ed94d71580d P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 03:04:12.290743 25445 consensus_queue.cc:1035] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260925 03:04:12.292376 24436 consensus_peers.cc:487] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 -> Peer 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979): Couldn't send request to peer 1bc1f2e62ab44af995165bd5116a9d43. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 367277daf8d04778b3fa4ed94d71580d. This is attempt 1: this message will repeat every 5th retry.
I20260925 03:04:12.293291 25445 consensus_queue.cc:1035] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260925 03:04:12.303103 25445 raft_consensus.cc:2932] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } } }
I20260925 03:04:12.306663 24542 raft_consensus.cc:2932] T 367277daf8d04778b3fa4ed94d71580d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } } }
I20260925 03:04:12.312997 24622 raft_consensus.cc:2932] T 367277daf8d04778b3fa4ed94d71580d P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } } }
I20260925 03:04:12.324570 24371 catalog_manager.cc:5495] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 reported cstate change: config changed from index -1 to 2, NON_VOTER 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) added. New cstate: current_term: 1 leader_uuid: "869b0a8575b54d08ae2817fad5dd42d5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260925 03:04:12.602962 25464 ts_tablet_manager.cc:927] T 29637f5a4a504ce6ba29033ccd0cd716 P 1bc1f2e62ab44af995165bd5116a9d43: Initiating tablet copy from peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:12.604898 25464 tablet_copy_client.cc:316] T 29637f5a4a504ce6ba29033ccd0cd716 P 1bc1f2e62ab44af995165bd5116a9d43: tablet copy: Beginning tablet copy session from remote peer at address 127.22.193.3:35105
I20260925 03:04:12.607220 24632 tablet_copy_service.cc:140] P eeab93abfdb141a895002ec9f312296c: Received BeginTabletCopySession request for tablet 29637f5a4a504ce6ba29033ccd0cd716 from peer 1bc1f2e62ab44af995165bd5116a9d43 ({username='slave'} at 127.0.0.1:34546)
I20260925 03:04:12.607751 24632 tablet_copy_service.cc:161] P eeab93abfdb141a895002ec9f312296c: Beginning new tablet copy session on tablet 29637f5a4a504ce6ba29033ccd0cd716 from peer 1bc1f2e62ab44af995165bd5116a9d43 at {username='slave'} at 127.0.0.1:34546: session id = 1bc1f2e62ab44af995165bd5116a9d43-29637f5a4a504ce6ba29033ccd0cd716
I20260925 03:04:12.616073 24632 tablet_copy_source_session.cc:205] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 03:04:12.619843 25464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29637f5a4a504ce6ba29033ccd0cd716. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:04:12.636636 25464 tablet_copy_client.cc:791] T 29637f5a4a504ce6ba29033ccd0cd716 P 1bc1f2e62ab44af995165bd5116a9d43: tablet copy: Starting download of 0 data blocks...
I20260925 03:04:12.637260 25464 tablet_copy_client.cc:655] T 29637f5a4a504ce6ba29033ccd0cd716 P 1bc1f2e62ab44af995165bd5116a9d43: tablet copy: Starting download of 1 WAL segments...
I20260925 03:04:12.643328 25464 tablet_copy_client.cc:531] T 29637f5a4a504ce6ba29033ccd0cd716 P 1bc1f2e62ab44af995165bd5116a9d43: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 03:04:12.651000 25464 tablet_bootstrap.cc:492] T 29637f5a4a504ce6ba29033ccd0cd716 P 1bc1f2e62ab44af995165bd5116a9d43: Bootstrap starting.
I20260925 03:04:12.670337 25466 ts_tablet_manager.cc:927] T 7e49248643df4131a1c42e74244e05c2 P 1bc1f2e62ab44af995165bd5116a9d43: Initiating tablet copy from peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187)
I20260925 03:04:12.672415 25464 tablet_bootstrap.cc:492] T 29637f5a4a504ce6ba29033ccd0cd716 P 1bc1f2e62ab44af995165bd5116a9d43: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 03:04:12.674065 25464 tablet_bootstrap.cc:492] T 29637f5a4a504ce6ba29033ccd0cd716 P 1bc1f2e62ab44af995165bd5116a9d43: Bootstrap complete.
I20260925 03:04:12.674849 25464 ts_tablet_manager.cc:1397] T 29637f5a4a504ce6ba29033ccd0cd716 P 1bc1f2e62ab44af995165bd5116a9d43: Time spent bootstrapping tablet: real 0.024s user 0.016s sys 0.007s
I20260925 03:04:12.677649 25464 raft_consensus.cc:348] T 29637f5a4a504ce6ba29033ccd0cd716 P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } }
I20260925 03:04:12.678627 25464 raft_consensus.cc:729] T 29637f5a4a504ce6ba29033ccd0cd716 P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1bc1f2e62ab44af995165bd5116a9d43, State: Initialized, Role: LEARNER
I20260925 03:04:12.685068 25464 consensus_queue.cc:260] T 29637f5a4a504ce6ba29033ccd0cd716 P 1bc1f2e62ab44af995165bd5116a9d43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } }
I20260925 03:04:12.697324 25469 ts_tablet_manager.cc:927] T e92f377a79354a6ebb4eec67303c13e3 P 1bc1f2e62ab44af995165bd5116a9d43: Initiating tablet copy from peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:12.702244 25469 tablet_copy_client.cc:316] T e92f377a79354a6ebb4eec67303c13e3 P 1bc1f2e62ab44af995165bd5116a9d43: tablet copy: Beginning tablet copy session from remote peer at address 127.22.193.3:35105
I20260925 03:04:12.703644 25466 tablet_copy_client.cc:316] T 7e49248643df4131a1c42e74244e05c2 P 1bc1f2e62ab44af995165bd5116a9d43: tablet copy: Beginning tablet copy session from remote peer at address 127.22.193.1:42187
I20260925 03:04:12.704293 24632 tablet_copy_service.cc:140] P eeab93abfdb141a895002ec9f312296c: Received BeginTabletCopySession request for tablet e92f377a79354a6ebb4eec67303c13e3 from peer 1bc1f2e62ab44af995165bd5116a9d43 ({username='slave'} at 127.0.0.1:34546)
I20260925 03:04:12.704880 24632 tablet_copy_service.cc:161] P eeab93abfdb141a895002ec9f312296c: Beginning new tablet copy session on tablet e92f377a79354a6ebb4eec67303c13e3 from peer 1bc1f2e62ab44af995165bd5116a9d43 at {username='slave'} at 127.0.0.1:34546: session id = 1bc1f2e62ab44af995165bd5116a9d43-e92f377a79354a6ebb4eec67303c13e3
I20260925 03:04:12.707633 24483 tablet_copy_service.cc:140] P 869b0a8575b54d08ae2817fad5dd42d5: Received BeginTabletCopySession request for tablet 7e49248643df4131a1c42e74244e05c2 from peer 1bc1f2e62ab44af995165bd5116a9d43 ({username='slave'} at 127.0.0.1:38920)
I20260925 03:04:12.708176 24483 tablet_copy_service.cc:161] P 869b0a8575b54d08ae2817fad5dd42d5: Beginning new tablet copy session on tablet 7e49248643df4131a1c42e74244e05c2 from peer 1bc1f2e62ab44af995165bd5116a9d43 at {username='slave'} at 127.0.0.1:38920: session id = 1bc1f2e62ab44af995165bd5116a9d43-7e49248643df4131a1c42e74244e05c2
I20260925 03:04:12.720348 24483 tablet_copy_source_session.cc:205] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 03:04:12.721397 25464 ts_tablet_manager.cc:1428] T 29637f5a4a504ce6ba29033ccd0cd716 P 1bc1f2e62ab44af995165bd5116a9d43: Time spent starting tablet: real 0.046s user 0.019s sys 0.011s
I20260925 03:04:12.722793 24632 tablet_copy_source_session.cc:205] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 03:04:12.725397 24631 tablet_copy_service.cc:342] P eeab93abfdb141a895002ec9f312296c: Request end of tablet copy session 1bc1f2e62ab44af995165bd5116a9d43-29637f5a4a504ce6ba29033ccd0cd716 received from {username='slave'} at 127.0.0.1:34546
I20260925 03:04:12.726171 24631 tablet_copy_service.cc:433] P eeab93abfdb141a895002ec9f312296c: ending tablet copy session 1bc1f2e62ab44af995165bd5116a9d43-29637f5a4a504ce6ba29033ccd0cd716 on tablet 29637f5a4a504ce6ba29033ccd0cd716 with peer 1bc1f2e62ab44af995165bd5116a9d43
I20260925 03:04:12.725667 25466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e49248643df4131a1c42e74244e05c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:04:12.736953 25469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e92f377a79354a6ebb4eec67303c13e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:04:12.741467 25466 tablet_copy_client.cc:791] T 7e49248643df4131a1c42e74244e05c2 P 1bc1f2e62ab44af995165bd5116a9d43: tablet copy: Starting download of 0 data blocks...
I20260925 03:04:12.742316 25466 tablet_copy_client.cc:655] T 7e49248643df4131a1c42e74244e05c2 P 1bc1f2e62ab44af995165bd5116a9d43: tablet copy: Starting download of 1 WAL segments...
I20260925 03:04:12.745935 25466 tablet_copy_client.cc:531] T 7e49248643df4131a1c42e74244e05c2 P 1bc1f2e62ab44af995165bd5116a9d43: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 03:04:12.754956 25469 tablet_copy_client.cc:791] T e92f377a79354a6ebb4eec67303c13e3 P 1bc1f2e62ab44af995165bd5116a9d43: tablet copy: Starting download of 0 data blocks...
I20260925 03:04:12.755743 25469 tablet_copy_client.cc:655] T e92f377a79354a6ebb4eec67303c13e3 P 1bc1f2e62ab44af995165bd5116a9d43: tablet copy: Starting download of 1 WAL segments...
I20260925 03:04:12.758142 25466 tablet_bootstrap.cc:492] T 7e49248643df4131a1c42e74244e05c2 P 1bc1f2e62ab44af995165bd5116a9d43: Bootstrap starting.
I20260925 03:04:12.762547 25469 tablet_copy_client.cc:531] T e92f377a79354a6ebb4eec67303c13e3 P 1bc1f2e62ab44af995165bd5116a9d43: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 03:04:12.772580 25469 tablet_bootstrap.cc:492] T e92f377a79354a6ebb4eec67303c13e3 P 1bc1f2e62ab44af995165bd5116a9d43: Bootstrap starting.
I20260925 03:04:12.787653 25466 tablet_bootstrap.cc:492] T 7e49248643df4131a1c42e74244e05c2 P 1bc1f2e62ab44af995165bd5116a9d43: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 03:04:12.788630 25466 tablet_bootstrap.cc:492] T 7e49248643df4131a1c42e74244e05c2 P 1bc1f2e62ab44af995165bd5116a9d43: Bootstrap complete.
I20260925 03:04:12.789389 25466 ts_tablet_manager.cc:1397] T 7e49248643df4131a1c42e74244e05c2 P 1bc1f2e62ab44af995165bd5116a9d43: Time spent bootstrapping tablet: real 0.031s user 0.023s sys 0.004s
I20260925 03:04:12.792243 25466 raft_consensus.cc:348] T 7e49248643df4131a1c42e74244e05c2 P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } }
I20260925 03:04:12.793334 25466 raft_consensus.cc:729] T 7e49248643df4131a1c42e74244e05c2 P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1bc1f2e62ab44af995165bd5116a9d43, State: Initialized, Role: LEARNER
I20260925 03:04:12.794085 25466 consensus_queue.cc:260] T 7e49248643df4131a1c42e74244e05c2 P 1bc1f2e62ab44af995165bd5116a9d43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } }
I20260925 03:04:12.796921 25466 ts_tablet_manager.cc:1428] T 7e49248643df4131a1c42e74244e05c2 P 1bc1f2e62ab44af995165bd5116a9d43: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260925 03:04:12.800969 24483 tablet_copy_service.cc:342] P 869b0a8575b54d08ae2817fad5dd42d5: Request end of tablet copy session 1bc1f2e62ab44af995165bd5116a9d43-7e49248643df4131a1c42e74244e05c2 received from {username='slave'} at 127.0.0.1:38920
I20260925 03:04:12.801370 24483 tablet_copy_service.cc:433] P 869b0a8575b54d08ae2817fad5dd42d5: ending tablet copy session 1bc1f2e62ab44af995165bd5116a9d43-7e49248643df4131a1c42e74244e05c2 on tablet 7e49248643df4131a1c42e74244e05c2 with peer 1bc1f2e62ab44af995165bd5116a9d43
I20260925 03:04:12.809635 25469 tablet_bootstrap.cc:492] T e92f377a79354a6ebb4eec67303c13e3 P 1bc1f2e62ab44af995165bd5116a9d43: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 03:04:12.810526 25469 tablet_bootstrap.cc:492] T e92f377a79354a6ebb4eec67303c13e3 P 1bc1f2e62ab44af995165bd5116a9d43: Bootstrap complete.
I20260925 03:04:12.811215 25469 ts_tablet_manager.cc:1397] T e92f377a79354a6ebb4eec67303c13e3 P 1bc1f2e62ab44af995165bd5116a9d43: Time spent bootstrapping tablet: real 0.039s user 0.019s sys 0.008s
I20260925 03:04:12.813524 25469 raft_consensus.cc:348] T e92f377a79354a6ebb4eec67303c13e3 P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } attrs { replace: true } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } }
I20260925 03:04:12.814366 25469 raft_consensus.cc:729] T e92f377a79354a6ebb4eec67303c13e3 P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1bc1f2e62ab44af995165bd5116a9d43, State: Initialized, Role: LEARNER
I20260925 03:04:12.814996 25469 consensus_queue.cc:260] T e92f377a79354a6ebb4eec67303c13e3 P 1bc1f2e62ab44af995165bd5116a9d43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } attrs { replace: true } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } }
I20260925 03:04:12.817274 25469 ts_tablet_manager.cc:1428] T e92f377a79354a6ebb4eec67303c13e3 P 1bc1f2e62ab44af995165bd5116a9d43: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260925 03:04:12.819038 24631 tablet_copy_service.cc:342] P eeab93abfdb141a895002ec9f312296c: Request end of tablet copy session 1bc1f2e62ab44af995165bd5116a9d43-e92f377a79354a6ebb4eec67303c13e3 received from {username='slave'} at 127.0.0.1:34546
I20260925 03:04:12.819406 24631 tablet_copy_service.cc:433] P eeab93abfdb141a895002ec9f312296c: ending tablet copy session 1bc1f2e62ab44af995165bd5116a9d43-e92f377a79354a6ebb4eec67303c13e3 on tablet e92f377a79354a6ebb4eec67303c13e3 with peer 1bc1f2e62ab44af995165bd5116a9d43
I20260925 03:04:12.833168 25469 ts_tablet_manager.cc:927] T 367277daf8d04778b3fa4ed94d71580d P 1bc1f2e62ab44af995165bd5116a9d43: Initiating tablet copy from peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187)
I20260925 03:04:12.834548 25469 tablet_copy_client.cc:316] T 367277daf8d04778b3fa4ed94d71580d P 1bc1f2e62ab44af995165bd5116a9d43: tablet copy: Beginning tablet copy session from remote peer at address 127.22.193.1:42187
I20260925 03:04:12.835872 24483 tablet_copy_service.cc:140] P 869b0a8575b54d08ae2817fad5dd42d5: Received BeginTabletCopySession request for tablet 367277daf8d04778b3fa4ed94d71580d from peer 1bc1f2e62ab44af995165bd5116a9d43 ({username='slave'} at 127.0.0.1:38920)
I20260925 03:04:12.836337 24483 tablet_copy_service.cc:161] P 869b0a8575b54d08ae2817fad5dd42d5: Beginning new tablet copy session on tablet 367277daf8d04778b3fa4ed94d71580d from peer 1bc1f2e62ab44af995165bd5116a9d43 at {username='slave'} at 127.0.0.1:38920: session id = 1bc1f2e62ab44af995165bd5116a9d43-367277daf8d04778b3fa4ed94d71580d
I20260925 03:04:12.842610 24483 tablet_copy_source_session.cc:205] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 03:04:12.845407 25469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 367277daf8d04778b3fa4ed94d71580d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:04:12.859293 25469 tablet_copy_client.cc:791] T 367277daf8d04778b3fa4ed94d71580d P 1bc1f2e62ab44af995165bd5116a9d43: tablet copy: Starting download of 0 data blocks...
I20260925 03:04:12.859813 25469 tablet_copy_client.cc:655] T 367277daf8d04778b3fa4ed94d71580d P 1bc1f2e62ab44af995165bd5116a9d43: tablet copy: Starting download of 1 WAL segments...
I20260925 03:04:12.863126 25469 tablet_copy_client.cc:531] T 367277daf8d04778b3fa4ed94d71580d P 1bc1f2e62ab44af995165bd5116a9d43: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 03:04:12.870363 25469 tablet_bootstrap.cc:492] T 367277daf8d04778b3fa4ed94d71580d P 1bc1f2e62ab44af995165bd5116a9d43: Bootstrap starting.
I20260925 03:04:12.887039 25469 tablet_bootstrap.cc:492] T 367277daf8d04778b3fa4ed94d71580d P 1bc1f2e62ab44af995165bd5116a9d43: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 03:04:12.887853 25469 tablet_bootstrap.cc:492] T 367277daf8d04778b3fa4ed94d71580d P 1bc1f2e62ab44af995165bd5116a9d43: Bootstrap complete.
I20260925 03:04:12.888479 25469 ts_tablet_manager.cc:1397] T 367277daf8d04778b3fa4ed94d71580d P 1bc1f2e62ab44af995165bd5116a9d43: Time spent bootstrapping tablet: real 0.018s user 0.010s sys 0.005s
I20260925 03:04:12.890892 25469 raft_consensus.cc:348] T 367277daf8d04778b3fa4ed94d71580d P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } }
I20260925 03:04:12.891764 25469 raft_consensus.cc:729] T 367277daf8d04778b3fa4ed94d71580d P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1bc1f2e62ab44af995165bd5116a9d43, State: Initialized, Role: LEARNER
I20260925 03:04:12.892453 25469 consensus_queue.cc:260] T 367277daf8d04778b3fa4ed94d71580d P 1bc1f2e62ab44af995165bd5116a9d43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } }
I20260925 03:04:12.894774 25469 ts_tablet_manager.cc:1428] T 367277daf8d04778b3fa4ed94d71580d P 1bc1f2e62ab44af995165bd5116a9d43: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260925 03:04:12.896605 24483 tablet_copy_service.cc:342] P 869b0a8575b54d08ae2817fad5dd42d5: Request end of tablet copy session 1bc1f2e62ab44af995165bd5116a9d43-367277daf8d04778b3fa4ed94d71580d received from {username='slave'} at 127.0.0.1:38920
I20260925 03:04:12.897011 24483 tablet_copy_service.cc:433] P 869b0a8575b54d08ae2817fad5dd42d5: ending tablet copy session 1bc1f2e62ab44af995165bd5116a9d43-367277daf8d04778b3fa4ed94d71580d on tablet 367277daf8d04778b3fa4ed94d71580d with peer 1bc1f2e62ab44af995165bd5116a9d43
I20260925 03:04:13.055101 25199 raft_consensus.cc:1203] T e92f377a79354a6ebb4eec67303c13e3 P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260925 03:04:13.121636 25199 raft_consensus.cc:1203] T 29637f5a4a504ce6ba29033ccd0cd716 P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260925 03:04:13.130172 25199 raft_consensus.cc:1203] T 7e49248643df4131a1c42e74244e05c2 P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260925 03:04:14.252888 24369 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:57656 (request call id 68) took 1127 ms. Trace:
I20260925 03:04:14.253333 24369 rpcz_store.cc:276] 0925 03:04:13.125927 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:13.126389 (+ 462us) service_pool.cc:227] Handling call
0925 03:04:14.252790 (+1126401us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":1}
I20260925 03:04:14.362501 24371 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39382 (request call id 179) took 1224 ms. Trace:
I20260925 03:04:14.369470 24371 rpcz_store.cc:276] 0925 03:04:13.137630 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:13.137827 (+ 197us) service_pool.cc:227] Handling call
0925 03:04:14.362425 (+1224598us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1196,"reported_tablets":1}
I20260925 03:04:14.365212 25486 raft_consensus.cc:482] T a4abe0ea757e4886be4945c5001e529a P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:14.374284 25486 raft_consensus.cc:504] T a4abe0ea757e4886be4945c5001e529a P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:14.399350 25486 leader_election.cc:288] T a4abe0ea757e4886be4945c5001e529a P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:14.380133 25489 raft_consensus.cc:482] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:14.400758 25498 raft_consensus.cc:482] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:14.402721 25500 raft_consensus.cc:482] T 11bb46bce3564841b7674e6a693bd2e6 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:14.404131 25480 raft_consensus.cc:482] T 21c43af5e6844fdaacc8c625fa01fd8d P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:14.404389 25494 raft_consensus.cc:482] T 37a157996e944db2906110d87ecb0c54 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:14.404508 25479 raft_consensus.cc:482] T edcf6e6a422d4d198d4fbb0f7d3d845d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:14.405493 25501 raft_consensus.cc:482] T 367277daf8d04778b3fa4ed94d71580d P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:14.406481 25495 raft_consensus.cc:482] T c85f5d1d3f004cf5817bb594fd8e6469 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:14.402730 25484 raft_consensus.cc:482] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:14.501142 25501 raft_consensus.cc:504] T 367277daf8d04778b3fa4ed94d71580d P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } }
I20260925 03:04:14.415403 25486 raft_consensus.cc:482] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:14.520809 25487 raft_consensus.cc:482] T 442d4a29b1d743599f4b6b4871e26668 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:14.496380 25480 raft_consensus.cc:504] T 21c43af5e6844fdaacc8c625fa01fd8d P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:14.495679 25489 raft_consensus.cc:504] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:14.496006 25479 raft_consensus.cc:504] T edcf6e6a422d4d198d4fbb0f7d3d845d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:14.497818 25498 raft_consensus.cc:504] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:14.525445 25486 raft_consensus.cc:504] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:14.431572 25500 raft_consensus.cc:504] T 11bb46bce3564841b7674e6a693bd2e6 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:14.516525 25495 raft_consensus.cc:504] T c85f5d1d3f004cf5817bb594fd8e6469 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:14.495882 25494 raft_consensus.cc:504] T 37a157996e944db2906110d87ecb0c54 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:14.516352 25484 raft_consensus.cc:504] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:14.557900 25483 raft_consensus.cc:482] T b1e85248d3b04b979dd080caeeb02fdb P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:14.621129 25502 raft_consensus.cc:482] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:14.621979 25486 leader_election.cc:288] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:14.632511 25486 raft_consensus.cc:482] T 04e068a0f80941dc9138cee063f53bab P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:14.621800 25502 raft_consensus.cc:504] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:14.695524 25502 leader_election.cc:288] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:14.673072 25489 leader_election.cc:288] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:14.672775 25486 raft_consensus.cc:504] T 04e068a0f80941dc9138cee063f53bab P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:14.585031 25487 raft_consensus.cc:504] T 442d4a29b1d743599f4b6b4871e26668 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:14.711436 25500 leader_election.cc:288] T 11bb46bce3564841b7674e6a693bd2e6 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:14.713064 25494 leader_election.cc:288] T 37a157996e944db2906110d87ecb0c54 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:14.714193 25480 leader_election.cc:288] T 21c43af5e6844fdaacc8c625fa01fd8d P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:14.715358 25479 leader_election.cc:288] T edcf6e6a422d4d198d4fbb0f7d3d845d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:14.716784 25479 raft_consensus.cc:482] T 0be4e0aa984f4f2b81a58e382e434ad5 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:14.717305 25479 raft_consensus.cc:504] T 0be4e0aa984f4f2b81a58e382e434ad5 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:14.673233 25501 leader_election.cc:288] T 367277daf8d04778b3fa4ed94d71580d P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:14.725713 25506 raft_consensus.cc:482] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:14.726603 25488 raft_consensus.cc:482] T 37a157996e944db2906110d87ecb0c54 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:14.726388 25506 raft_consensus.cc:504] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:14.733214 25488 raft_consensus.cc:504] T 37a157996e944db2906110d87ecb0c54 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:14.702816 25477 raft_consensus.cc:482] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:14.714071 25492 raft_consensus.cc:482] T f22ddd7d1c444dc58f45a1fb0b3e02bc P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:14.738792 25506 leader_election.cc:288] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:14.739176 25488 leader_election.cc:288] T 37a157996e944db2906110d87ecb0c54 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:14.740581 25497 raft_consensus.cc:482] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:14.740911 25487 leader_election.cc:288] T 442d4a29b1d743599f4b6b4871e26668 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:14.617443 25483 raft_consensus.cc:504] T b1e85248d3b04b979dd080caeeb02fdb P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:14.743577 25486 leader_election.cc:288] T 04e068a0f80941dc9138cee063f53bab P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:14.745549 25498 leader_election.cc:288] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:14.746403 25495 leader_election.cc:288] T c85f5d1d3f004cf5817bb594fd8e6469 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:14.747583 25490 raft_consensus.cc:482] T b4ebd2b043024a548005c8b6b00b0449 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:14.759585 25486 raft_consensus.cc:482] T 71f461a6e2ff434993b78a2d077f0abd P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:14.776209 25486 raft_consensus.cc:504] T 71f461a6e2ff434993b78a2d077f0abd P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:14.787508 25479 leader_election.cc:288] T 0be4e0aa984f4f2b81a58e382e434ad5 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:14.788734 25486 leader_election.cc:288] T 71f461a6e2ff434993b78a2d077f0abd P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:14.789227 25486 raft_consensus.cc:482] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:14.795405 25505 raft_consensus.cc:482] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:14.796073 25505 raft_consensus.cc:504] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:14.798406 25505 leader_election.cc:288] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:14.798986 25505 raft_consensus.cc:482] T 550118bc1a0641ce9803bdbf43f2069f P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:14.799466 25505 raft_consensus.cc:504] T 550118bc1a0641ce9803bdbf43f2069f P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:14.801226 25504 raft_consensus.cc:482] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:14.801793 25504 raft_consensus.cc:504] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:14.803992 25504 leader_election.cc:288] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:14.806560 25480 raft_consensus.cc:482] T 43366a0a2f9d4344bf4e37ffd290ef8a P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:14.789660 25486 raft_consensus.cc:504] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:14.807255 25480 raft_consensus.cc:504] T 43366a0a2f9d4344bf4e37ffd290ef8a P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:14.809557 25500 raft_consensus.cc:482] T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:14.809166 25480 leader_election.cc:288] T 43366a0a2f9d4344bf4e37ffd290ef8a P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:14.804533 25504 raft_consensus.cc:482] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:14.809574 25486 leader_election.cc:288] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:14.770618 25497 raft_consensus.cc:504] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:14.749948 25492 raft_consensus.cc:504] T f22ddd7d1c444dc58f45a1fb0b3e02bc P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:14.820003 25501 raft_consensus.cc:482] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:14.820551 25494 raft_consensus.cc:482] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:14.840435 25486 raft_consensus.cc:482] T b4ebd2b043024a548005c8b6b00b0449 P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:14.842033 25486 raft_consensus.cc:504] T b4ebd2b043024a548005c8b6b00b0449 P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:14.842500 25518 raft_consensus.cc:482] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:14.843063 25518 raft_consensus.cc:504] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:14.827068 25480 raft_consensus.cc:482] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:14.852674 25480 raft_consensus.cc:504] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:14.838536 25484 leader_election.cc:288] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:14.848531 25501 raft_consensus.cc:504] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:14.854694 25480 leader_election.cc:288] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:14.841733 25494 raft_consensus.cc:504] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:14.855890 25501 leader_election.cc:288] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:14.856441 25494 leader_election.cc:288] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:14.744410 25477 raft_consensus.cc:504] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:14.810101 25500 raft_consensus.cc:504] T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:14.836469 25504 raft_consensus.cc:504] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:14.784862 25490 raft_consensus.cc:504] T b4ebd2b043024a548005c8b6b00b0449 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:14.860680 25500 leader_election.cc:288] T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:14.858857 25495 raft_consensus.cc:482] T 420163a6cdce412ebc21c25fe3835638 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:14.861459 25490 leader_election.cc:288] T b4ebd2b043024a548005c8b6b00b0449 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:14.862010 25486 leader_election.cc:288] T b4ebd2b043024a548005c8b6b00b0449 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:14.862597 25483 leader_election.cc:288] T b1e85248d3b04b979dd080caeeb02fdb P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:14.864159 25505 leader_election.cc:288] T 550118bc1a0641ce9803bdbf43f2069f P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:14.884325 25486 raft_consensus.cc:482] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:14.900100 25505 raft_consensus.cc:482] T 3d89396576884f7c8901dba54f783947 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:14.906316 25483 raft_consensus.cc:482] T e6b6ec010caf4689b8c90449764f5cf9 P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:14.904639 25492 leader_election.cc:288] T f22ddd7d1c444dc58f45a1fb0b3e02bc P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:14.916427 25491 raft_consensus.cc:482] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:14.917119 25512 raft_consensus.cc:482] T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:14.917738 25514 raft_consensus.cc:482] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:14.910537 25477 leader_election.cc:288] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:14.932111 25510 raft_consensus.cc:482] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:14.933470 25510 raft_consensus.cc:504] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:14.918386 25508 raft_consensus.cc:482] T 21c43af5e6844fdaacc8c625fa01fd8d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:14.903070 25486 raft_consensus.cc:504] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:14.939850 25504 leader_election.cc:288] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:14.914641 25490 raft_consensus.cc:482] T 8e5819c15eac4544a57e5ee6c9d4f117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:14.883767 25495 raft_consensus.cc:504] T 420163a6cdce412ebc21c25fe3835638 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:14.946444 25525 raft_consensus.cc:482] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:14.911562 25517 raft_consensus.cc:482] T 8ac466793d094325ace224708b2158ed P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:14.942461 25502 raft_consensus.cc:482] T e6b6ec010caf4689b8c90449764f5cf9 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:14.942277 25480 raft_consensus.cc:482] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:14.913509 25505 raft_consensus.cc:504] T 3d89396576884f7c8901dba54f783947 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:14.938687 25497 leader_election.cc:288] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:14.925005 25501 raft_consensus.cc:482] T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:14.940004 25527 raft_consensus.cc:482] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:14.930410 25512 raft_consensus.cc:504] T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:14.924327 25483 raft_consensus.cc:504] T e6b6ec010caf4689b8c90449764f5cf9 P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:14.985560 25507 raft_consensus.cc:482] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:14.955958 25500 raft_consensus.cc:482] T edcf6e6a422d4d198d4fbb0f7d3d845d P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:14.953949 25504 raft_consensus.cc:482] T 48ca6ccd98194cf2af7b5278d15bf4ea P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:14.985842 25485 raft_consensus.cc:482] T 3d89396576884f7c8901dba54f783947 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:14.986348 25515 raft_consensus.cc:482] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:14.965255 25488 raft_consensus.cc:482] T 293ef69caf514ac98510a68495f54eda P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:14.926471 25491 raft_consensus.cc:504] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:14.975039 25494 raft_consensus.cc:482] T 71f461a6e2ff434993b78a2d077f0abd P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:14.985575 25493 raft_consensus.cc:482] T c88cbd39e01f44628a094af516be2833 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:14.963660 25516 raft_consensus.cc:482] T 8e5819c15eac4544a57e5ee6c9d4f117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:15.007897 25503 raft_consensus.cc:482] T b030d84c670b48d5bff337ccc0c77626 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:14.994362 25518 leader_election.cc:288] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:14.956817 25490 raft_consensus.cc:504] T 8e5819c15eac4544a57e5ee6c9d4f117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:14.992493 25489 raft_consensus.cc:482] T 7b8e0d260a884ecdb20f79de46c36b19 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:15.043653 25521 raft_consensus.cc:482] T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:15.025475 25530 raft_consensus.cc:482] T 100c113a9fd84cc8adf2ec7b33ea0093 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:14.994405 25487 raft_consensus.cc:482] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:14.980983 25480 raft_consensus.cc:504] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:14.987300 25502 raft_consensus.cc:504] T e6b6ec010caf4689b8c90449764f5cf9 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:15.016199 25498 raft_consensus.cc:482] T a7aa47c48404465b94a36fd819fe3c7a P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:15.059134 25513 raft_consensus.cc:482] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:14.997344 25500 raft_consensus.cc:504] T edcf6e6a422d4d198d4fbb0f7d3d845d P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:15.045536 25479 raft_consensus.cc:482] T 1414dcfe53f64a78ae413531eb8ec117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:15.065228 25531 raft_consensus.cc:482] T 04e068a0f80941dc9138cee063f53bab P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:14.992908 25501 raft_consensus.cc:504] T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } }
I20260925 03:04:15.049324 25484 raft_consensus.cc:482] T 06f4a7eaf6534e4986c9fd35911038e4 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:15.026180 25515 raft_consensus.cc:504] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } }
I20260925 03:04:15.013772 25504 raft_consensus.cc:504] T 48ca6ccd98194cf2af7b5278d15bf4ea P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:15.082751 25483 leader_election.cc:288] T e6b6ec010caf4689b8c90449764f5cf9 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:15.015292 25488 raft_consensus.cc:504] T 293ef69caf514ac98510a68495f54eda P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:15.058615 25510 leader_election.cc:288] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:14.954584 25508 raft_consensus.cc:504] T 21c43af5e6844fdaacc8c625fa01fd8d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:15.042322 25494 raft_consensus.cc:504] T 71f461a6e2ff434993b78a2d077f0abd P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:15.069355 25477 raft_consensus.cc:482] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:15.086040 25499 raft_consensus.cc:482] T 43366a0a2f9d4344bf4e37ffd290ef8a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:14.949280 25514 raft_consensus.cc:504] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:15.066121 25487 raft_consensus.cc:504] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:15.083391 25489 raft_consensus.cc:504] T 7b8e0d260a884ecdb20f79de46c36b19 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:15.123684 25508 leader_election.cc:288] T 21c43af5e6844fdaacc8c625fa01fd8d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:15.085963 25495 leader_election.cc:288] T 420163a6cdce412ebc21c25fe3835638 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:15.130650 25486 leader_election.cc:288] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:15.102908 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e54ca1757ee4e42956a9cebbeca4f62" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:15.044802 25485 raft_consensus.cc:504] T 3d89396576884f7c8901dba54f783947 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:15.111696 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1e85248d3b04b979dd080caeeb02fdb" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:15.133430 25477 raft_consensus.cc:504] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:15.090256 24467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c81a1b3b616492ab6c964f45af36077" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:15.141129 24467 raft_consensus.cc:2454] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 1.
I20260925 03:04:15.138211 25198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c81a1b3b616492ab6c964f45af36077" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:15.141896 25198 raft_consensus.cc:2454] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 1.
I20260925 03:04:15.142982 25512 leader_election.cc:288] T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:15.143556 25512 raft_consensus.cc:482] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:15.144001 25512 raft_consensus.cc:504] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:15.091519 25479 raft_consensus.cc:504] T 1414dcfe53f64a78ae413531eb8ec117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:14.980440 25517 raft_consensus.cc:504] T 8ac466793d094325ace224708b2158ed P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:15.083822 25498 raft_consensus.cc:504] T a7aa47c48404465b94a36fd819fe3c7a P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:15.012777 25527 raft_consensus.cc:504] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } }
I20260925 03:04:15.148976 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c81a1b3b616492ab6c964f45af36077" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:15.046581 25507 raft_consensus.cc:504] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:15.125252 25491 leader_election.cc:288] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:15.005901 25525 raft_consensus.cc:504] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:15.161979 24464 raft_consensus.cc:2454] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 1.
I20260925 03:04:15.111735 25484 raft_consensus.cc:504] T 06f4a7eaf6534e4986c9fd35911038e4 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:15.115768 25531 raft_consensus.cc:504] T 04e068a0f80941dc9138cee063f53bab P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:15.111285 25513 raft_consensus.cc:504] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } }
I20260925 03:04:15.102047 25511 raft_consensus.cc:482] T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:15.159063 25481 raft_consensus.cc:482] T 8ac466793d094325ace224708b2158ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:15.151948 25505 leader_election.cc:288] T 3d89396576884f7c8901dba54f783947 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:15.069609 25493 raft_consensus.cc:504] T c88cbd39e01f44628a094af516be2833 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:15.057097 25503 raft_consensus.cc:504] T b030d84c670b48d5bff337ccc0c77626 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:15.170696 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37a157996e944db2906110d87ecb0c54" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:15.058308 25516 raft_consensus.cc:504] T 8e5819c15eac4544a57e5ee6c9d4f117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:15.173151 24617 raft_consensus.cc:2454] T 37a157996e944db2906110d87ecb0c54 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 1.
I20260925 03:04:15.133024 25499 raft_consensus.cc:504] T 43366a0a2f9d4344bf4e37ffd290ef8a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:15.179201 25198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4ebd2b043024a548005c8b6b00b0449" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:15.189030 25198 raft_consensus.cc:2454] T b4ebd2b043024a548005c8b6b00b0449 P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 1.
I20260925 03:04:15.093227 25521 raft_consensus.cc:504] T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:15.190526 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442d4a29b1d743599f4b6b4871e26668" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:15.191187 24617 raft_consensus.cc:2454] T 442d4a29b1d743599f4b6b4871e26668 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 1.
I20260925 03:04:15.146708 25495 raft_consensus.cc:482] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:15.229259 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4ebd2b043024a548005c8b6b00b0449" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:15.235594 24468 raft_consensus.cc:2454] T b4ebd2b043024a548005c8b6b00b0449 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 1.
I20260925 03:04:15.239110 25525 leader_election.cc:288] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
W20260925 03:04:15.240837 24402 debug-util.cc:398] Leaking SignalData structure 0x7b080003d660 after lost signal to thread 23303
I20260925 03:04:15.244364 24614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35992316cb8d4c028333388eff02ecde" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:15.245047 24614 raft_consensus.cc:2454] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 1.
I20260925 03:04:15.075155 25530 raft_consensus.cc:504] T 100c113a9fd84cc8adf2ec7b33ea0093 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:15.177742 25490 leader_election.cc:288] T 8e5819c15eac4544a57e5ee6c9d4f117 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:15.192796 25509 raft_consensus.cc:482] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:15.191236 25502 leader_election.cc:288] T e6b6ec010caf4689b8c90449764f5cf9 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:15.229528 25500 leader_election.cc:288] T edcf6e6a422d4d198d4fbb0f7d3d845d P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:15.186924 25516 leader_election.cc:288] T 8e5819c15eac4544a57e5ee6c9d4f117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
W20260925 03:04:15.250675 24402 debug-util.cc:398] Leaking SignalData structure 0x7b080008eba0 after lost signal to thread 24405
I20260925 03:04:15.229965 25482 raft_consensus.cc:482] T 4e5332a5d329493991dff994e70ea92c P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:15.231294 25479 leader_election.cc:288] T 1414dcfe53f64a78ae413531eb8ec117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:15.232452 25498 leader_election.cc:288] T a7aa47c48404465b94a36fd819fe3c7a P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:15.230219 25523 raft_consensus.cc:482] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:15.228922 25494 leader_election.cc:288] T 71f461a6e2ff434993b78a2d077f0abd P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:15.229305 25489 leader_election.cc:288] T 7b8e0d260a884ecdb20f79de46c36b19 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:15.253150 25480 leader_election.cc:288] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:15.184801 25499 leader_election.cc:288] T 43366a0a2f9d4344bf4e37ffd290ef8a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:15.253690 25497 raft_consensus.cc:482] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:15.183447 25493 leader_election.cc:288] T c88cbd39e01f44628a094af516be2833 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:15.181530 25503 leader_election.cc:288] T b030d84c670b48d5bff337ccc0c77626 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:15.177214 25533 raft_consensus.cc:482] T 74443a0a1dc34d88a2f42e3b722db405 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:15.177232 25524 raft_consensus.cc:482] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:15.260442 25504 leader_election.cc:288] T 48ca6ccd98194cf2af7b5278d15bf4ea P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:15.262403 25485 leader_election.cc:288] T 3d89396576884f7c8901dba54f783947 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:15.276644 25198 raft_consensus.cc:1203] T 367277daf8d04778b3fa4ed94d71580d P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260925 03:04:15.273141 24614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b5df303cd6a4d59aca5c18bbbff24ed" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:15.280934 24614 raft_consensus.cc:2454] T 0b5df303cd6a4d59aca5c18bbbff24ed P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 1.
I20260925 03:04:15.259348 25501 leader_election.cc:288] T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:15.267139 25527 leader_election.cc:288] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:15.263505 25529 raft_consensus.cc:482] T 11bb46bce3564841b7674e6a693bd2e6 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:15.251722 25495 raft_consensus.cc:504] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:15.274969 25488 leader_election.cc:288] T 293ef69caf514ac98510a68495f54eda P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:15.284472 25513 leader_election.cc:288] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:15.229123 25481 raft_consensus.cc:504] T 8ac466793d094325ace224708b2158ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:15.187526 25511 raft_consensus.cc:504] T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:04:15.268841 24402 debug-util.cc:398] Leaking SignalData structure 0x7b080008ec60 after lost signal to thread 24418
I20260925 03:04:15.277036 25487 leader_election.cc:288] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:15.318589 25514 leader_election.cc:288] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:15.287744 25529 raft_consensus.cc:504] T 11bb46bce3564841b7674e6a693bd2e6 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:15.279587 25516 raft_consensus.cc:482] T 25102ba08e4e49388f6176badcd10a25 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:15.279841 25498 raft_consensus.cc:482] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:15.276899 25517 leader_election.cc:288] T 8ac466793d094325ace224708b2158ed P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:15.270251 25523 raft_consensus.cc:504] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:15.288524 25477 leader_election.cc:288] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:15.283414 25512 leader_election.cc:288] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:15.281764 25497 raft_consensus.cc:504] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:15.359102 25531 leader_election.cc:288] T 04e068a0f80941dc9138cee063f53bab P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:15.359570 24618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f68804eb5a244d87bb395d50683c6012" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:15.360411 25496 raft_consensus.cc:482] T 367277daf8d04778b3fa4ed94d71580d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:15.364187 25529 leader_election.cc:288] T 11bb46bce3564841b7674e6a693bd2e6 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:15.340950 25484 leader_election.cc:288] T 06f4a7eaf6534e4986c9fd35911038e4 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:15.285389 25482 raft_consensus.cc:504] T 4e5332a5d329493991dff994e70ea92c P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:15.376114 25482 leader_election.cc:288] T 4e5332a5d329493991dff994e70ea92c P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:15.285704 25509 raft_consensus.cc:504] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:15.379407 25521 leader_election.cc:288] T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:15.351755 25508 raft_consensus.cc:482] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:15.391203 24614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abd5b8dc1b7e49e3aa9af900d73ff0e7" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:15.391924 24614 raft_consensus.cc:2454] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 1.
I20260925 03:04:15.396574 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367277daf8d04778b3fa4ed94d71580d" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:15.365729 25506 raft_consensus.cc:482] T 420163a6cdce412ebc21c25fe3835638 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:15.343014 25498 raft_consensus.cc:504] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:15.362198 25507 leader_election.cc:288] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:15.355729 25515 leader_election.cc:288] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:15.357803 25516 raft_consensus.cc:504] T 25102ba08e4e49388f6176badcd10a25 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:15.414938 25199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6b6ec010caf4689b8c90449764f5cf9" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
W20260925 03:04:15.392861 24402 debug-util.cc:398] Leaking SignalData structure 0x7b080031a700 after lost signal to thread 24497
I20260925 03:04:15.402082 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bb46bce3564841b7674e6a693bd2e6" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:15.417299 24545 raft_consensus.cc:2454] T 11bb46bce3564841b7674e6a693bd2e6 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 1.
I20260925 03:04:15.388633 25503 raft_consensus.cc:482] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:15.350736 25523 leader_election.cc:288] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:15.393653 25521 raft_consensus.cc:482] T cc2df40db08649c9a75e90000b878559 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:15.415844 25199 raft_consensus.cc:2454] T e6b6ec010caf4689b8c90449764f5cf9 P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 1.
I20260925 03:04:15.393541 25485 raft_consensus.cc:482] T ac9357d144aa49789faaf31fe5e6bf89 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:15.416384 24614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e5819c15eac4544a57e5ee6c9d4f117" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:15.287679 25524 raft_consensus.cc:504] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:15.297940 25533 raft_consensus.cc:504] T 74443a0a1dc34d88a2f42e3b722db405 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:15.420238 25487 raft_consensus.cc:482] T 25102ba08e4e49388f6176badcd10a25 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:15.437584 25523 raft_consensus.cc:482] T b627ecb4a5384ef392aa695e06599c39 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:15.437422 25487 raft_consensus.cc:504] T 25102ba08e4e49388f6176badcd10a25 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:15.400027 25477 raft_consensus.cc:482] T 7993f9c3e2a44f408bbf5d1e395e2306 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:15.435979 25490 raft_consensus.cc:482] T e92f377a79354a6ebb4eec67303c13e3 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:15.436393 25488 raft_consensus.cc:482] T 9e8af2641e124f9589c5ccfdac785f63 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:15.435497 25513 raft_consensus.cc:482] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:15.436731 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37a157996e944db2906110d87ecb0c54" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:15.396055 25508 raft_consensus.cc:504] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:15.438061 25523 raft_consensus.cc:504] T b627ecb4a5384ef392aa695e06599c39 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:15.456857 25489 raft_consensus.cc:482] T f22ddd7d1c444dc58f45a1fb0b3e02bc P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:15.431531 25503 raft_consensus.cc:504] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:15.457425 25489 raft_consensus.cc:504] T f22ddd7d1c444dc58f45a1fb0b3e02bc P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:15.458772 25513 raft_consensus.cc:504] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:15.459235 25482 raft_consensus.cc:482] T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:15.459374 24618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b8e0d260a884ecdb20f79de46c36b19" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:15.459810 25482 raft_consensus.cc:504] T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:15.460259 24618 raft_consensus.cc:2454] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 1.
I20260925 03:04:15.454663 25488 raft_consensus.cc:504] T 9e8af2641e124f9589c5ccfdac785f63 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:15.465052 24614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5f9b503b0ab43999d582015a367441e" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:15.465806 25522 raft_consensus.cc:482] T a7aa47c48404465b94a36fd819fe3c7a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:15.406188 25506 raft_consensus.cc:504] T 420163a6cdce412ebc21c25fe3835638 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:15.473803 25522 raft_consensus.cc:504] T a7aa47c48404465b94a36fd819fe3c7a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:15.394291 25496 raft_consensus.cc:504] T 367277daf8d04778b3fa4ed94d71580d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } }
I20260925 03:04:15.456564 25537 raft_consensus.cc:482] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:15.449905 25521 raft_consensus.cc:504] T cc2df40db08649c9a75e90000b878559 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:15.452487 25477 raft_consensus.cc:504] T 7993f9c3e2a44f408bbf5d1e395e2306 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:15.452139 25485 raft_consensus.cc:504] T ac9357d144aa49789faaf31fe5e6bf89 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:15.568341 25198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e54ca1757ee4e42956a9cebbeca4f62" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:15.603072 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f1fac028a144058b92331c0d9b44ebf" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:15.631511 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c85f5d1d3f004cf5817bb594fd8e6469" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:15.576987 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c88cbd39e01f44628a094af516be2833" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:15.632326 24545 raft_consensus.cc:2454] T c85f5d1d3f004cf5817bb594fd8e6469 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 1.
I20260925 03:04:15.633791 25198 raft_consensus.cc:2454] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 1.
I20260925 03:04:15.632896 24547 raft_consensus.cc:2454] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 1.
I20260925 03:04:15.458344 25490 raft_consensus.cc:504] T e92f377a79354a6ebb4eec67303c13e3 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } attrs { replace: true } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } }
W20260925 03:04:15.633967 24511 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.4:43979, user_credentials={real_user=slave}} blocked reactor thread for 58992.4us
I20260925 03:04:15.638043 24615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9c33fd100ff44c1a3247319a324a044" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:15.566761 25501 raft_consensus.cc:482] T a4abe0ea757e4886be4945c5001e529a P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
W20260925 03:04:15.638083 24588 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.589s user 0.040s sys 0.091s
I20260925 03:04:15.644352 24615 raft_consensus.cc:2454] T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 1.
I20260925 03:04:15.608887 25535 raft_consensus.cc:482] T c88cbd39e01f44628a094af516be2833 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:15.607496 25482 leader_election.cc:288] T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
W20260925 03:04:15.652218 24588 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.604s user 0.041s sys 0.092s
I20260925 03:04:15.535545 25197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "293ef69caf514ac98510a68495f54eda" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:15.653785 24543 raft_consensus.cc:2454] T 367277daf8d04778b3fa4ed94d71580d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 1.
I20260925 03:04:15.655939 24615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:15.642318 25523 leader_election.cc:288] T b627ecb4a5384ef392aa695e06599c39 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:15.571843 25522 leader_election.cc:288] T a7aa47c48404465b94a36fd819fe3c7a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:15.485455 25509 leader_election.cc:288] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
W20260925 03:04:15.642619 24402 debug-util.cc:398] Leaking SignalData structure 0x7b08001f0940 after lost signal to thread 24572
I20260925 03:04:15.654803 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "293ef69caf514ac98510a68495f54eda" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:15.644892 25529 raft_consensus.cc:482] T 550118bc1a0641ce9803bdbf43f2069f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:15.573094 25511 leader_election.cc:288] T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:15.699605 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15861cd28e354598a3c6bfc4337c34bb" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:15.643571 25487 leader_election.cc:288] T 25102ba08e4e49388f6176badcd10a25 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:15.700277 24467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6b6ec010caf4689b8c90449764f5cf9" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:15.709072 24467 raft_consensus.cc:2454] T e6b6ec010caf4689b8c90449764f5cf9 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 1.
I20260925 03:04:15.701557 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5aec38e66a947c681e6c2a72b05e6ca" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:15.656209 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7691c6b475428c82d6181d693d26f3" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:15.710093 24622 raft_consensus.cc:2454] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 1.
I20260925 03:04:15.703486 24468 raft_consensus.cc:2454] T 293ef69caf514ac98510a68495f54eda P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 1.
I20260925 03:04:15.640264 25524 leader_election.cc:288] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:15.644455 25506 leader_election.cc:288] T 420163a6cdce412ebc21c25fe3835638 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:15.641326 25490 leader_election.cc:288] T e92f377a79354a6ebb4eec67303c13e3 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:15.635414 25495 leader_election.cc:288] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:15.641180 25489 leader_election.cc:288] T f22ddd7d1c444dc58f45a1fb0b3e02bc P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:15.696795 25540 raft_consensus.cc:482] T 48ca6ccd98194cf2af7b5278d15bf4ea P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:15.711874 25165 leader_election.cc:302] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5; no voters:
I20260925 03:04:15.711493 25542 raft_consensus.cc:482] T 100c113a9fd84cc8adf2ec7b33ea0093 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:15.712212 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21c43af5e6844fdaacc8c625fa01fd8d" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:15.713056 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e54ca1757ee4e42956a9cebbeca4f62" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:15.713403 25531 raft_consensus.cc:482] T 7993f9c3e2a44f408bbf5d1e395e2306 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:15.713893 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5332a5d329493991dff994e70ea92c" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:15.715962 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06f4a7eaf6534e4986c9fd35911038e4" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:15.740125 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367277daf8d04778b3fa4ed94d71580d" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:15.717136 24615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60d45b7d467d4e1ab92367f6d7d52b15" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:15.720953 24436 leader_election.cc:302] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5; no voters:
I20260925 03:04:15.724041 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35992316cb8d4c028333388eff02ecde" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:15.711088 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c23337a7cb74184a0e57b7095cb6a11" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:15.703840 25498 leader_election.cc:288] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:15.706900 25505 raft_consensus.cc:482] T 5278a60f97da4e10aecd50a8d9a9a43c P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:15.709758 25508 leader_election.cc:288] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:15.742985 24622 raft_consensus.cc:2454] T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 1.
I20260925 03:04:15.650483 25537 raft_consensus.cc:504] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:15.743479 24615 raft_consensus.cc:2454] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 1.
I20260925 03:04:15.744481 24436 leader_election.cc:302] T b4ebd2b043024a548005c8b6b00b0449 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5; no voters:
I20260925 03:04:15.720942 25497 leader_election.cc:288] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:15.746595 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bb46bce3564841b7674e6a693bd2e6" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:15.747159 24620 raft_consensus.cc:2454] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 1.
I20260925 03:04:15.711687 25536 raft_consensus.cc:482] T c85f5d1d3f004cf5817bb594fd8e6469 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:15.745257 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35992316cb8d4c028333388eff02ecde" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:15.653298 25501 raft_consensus.cc:504] T a4abe0ea757e4886be4945c5001e529a P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:15.726176 25516 leader_election.cc:288] T 25102ba08e4e49388f6176badcd10a25 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:15.726069 25503 leader_election.cc:288] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:15.725862 25481 leader_election.cc:288] T 8ac466793d094325ace224708b2158ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:15.737751 25513 leader_election.cc:288] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:15.678087 25535 raft_consensus.cc:504] T c88cbd39e01f44628a094af516be2833 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:15.745198 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f22ddd7d1c444dc58f45a1fb0b3e02bc" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:15.715989 25529 raft_consensus.cc:504] T 550118bc1a0641ce9803bdbf43f2069f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:15.731451 25540 raft_consensus.cc:504] T 48ca6ccd98194cf2af7b5278d15bf4ea P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:15.853307 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4abe0ea757e4886be4945c5001e529a" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:15.814349 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceab6b0fccd54e909a64f66fe81a9a1f" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:15.813776 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37a157996e944db2906110d87ecb0c54" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:15.844462 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37a157996e944db2906110d87ecb0c54" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:15.764178 24467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcf6e6a422d4d198d4fbb0f7d3d845d" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:15.827185 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21c43af5e6844fdaacc8c625fa01fd8d" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
W20260925 03:04:15.830545 24436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.4:43979, user_credentials={real_user=slave}} blocked reactor thread for 86128.9us
I20260925 03:04:15.806546 24614 raft_consensus.cc:2454] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 1.
I20260925 03:04:15.885747 24465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be4e0aa984f4f2b81a58e382e434ad5" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:15.813957 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43366a0a2f9d4344bf4e37ffd290ef8a" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:15.886194 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f68804eb5a244d87bb395d50683c6012" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:15.894825 24542 raft_consensus.cc:2454] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 1.
I20260925 03:04:15.745560 25533 leader_election.cc:288] T 74443a0a1dc34d88a2f42e3b722db405 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:15.887063 25537 leader_election.cc:288] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:15.887188 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71f461a6e2ff434993b78a2d077f0abd" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:15.887678 25496 leader_election.cc:288] T 367277daf8d04778b3fa4ed94d71580d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:15.747359 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e725b82985746b39c0cc8f671c95353" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:15.887784 25529 leader_election.cc:288] T 550118bc1a0641ce9803bdbf43f2069f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:15.888051 24616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b627ecb4a5384ef392aa695e06599c39" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:15.901607 24616 raft_consensus.cc:2454] T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 1.
I20260925 03:04:15.902073 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "420163a6cdce412ebc21c25fe3835638" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:15.902374 24613 raft_consensus.cc:2454] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 1.
I20260925 03:04:15.902733 24541 raft_consensus.cc:2454] T 420163a6cdce412ebc21c25fe3835638 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 1.
I20260925 03:04:15.888557 24617 raft_consensus.cc:2454] T c88cbd39e01f44628a094af516be2833 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 1.
I20260925 03:04:15.888980 25521 leader_election.cc:288] T cc2df40db08649c9a75e90000b878559 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:15.883201 25477 leader_election.cc:288] T 7993f9c3e2a44f408bbf5d1e395e2306 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
W20260925 03:04:15.744581 24402 debug-util.cc:398] Leaking SignalData structure 0x7b08002d34e0 after lost signal to thread 24646
I20260925 03:04:15.891721 24436 leader_election.cc:302] T e6b6ec010caf4689b8c90449764f5cf9 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5; no voters:
I20260925 03:04:15.885633 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f1fac028a144058b92331c0d9b44ebf" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:15.892738 24615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e92f377a79354a6ebb4eec67303c13e3" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:15.892861 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b627ecb4a5384ef392aa695e06599c39" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:15.744336 25488 leader_election.cc:288] T 9e8af2641e124f9589c5ccfdac785f63 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:15.762742 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74443a0a1dc34d88a2f42e3b722db405" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:15.762272 25517 raft_consensus.cc:2781] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 03:04:15.805629 25490 raft_consensus.cc:2781] T b4ebd2b043024a548005c8b6b00b0449 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 03:04:15.749140 25489 raft_consensus.cc:2781] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 03:04:15.896754 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b030d84c670b48d5bff337ccc0c77626" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:15.815790 25485 leader_election.cc:288] T ac9357d144aa49789faaf31fe5e6bf89 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:15.895687 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c85f5d1d3f004cf5817bb594fd8e6469" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:15.898728 24465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35992316cb8d4c028333388eff02ecde" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:15.898156 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43366a0a2f9d4344bf4e37ffd290ef8a" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
W20260925 03:04:15.750628 25165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 104706us
I20260925 03:04:15.891867 25535 leader_election.cc:288] T c88cbd39e01f44628a094af516be2833 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:15.816214 24539 raft_consensus.cc:2454] T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 1.
I20260925 03:04:15.905025 24467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "550118bc1a0641ce9803bdbf43f2069f" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:15.912228 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5aec38e66a947c681e6c2a72b05e6ca" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:15.908089 25539 raft_consensus.cc:2781] T e6b6ec010caf4689b8c90449764f5cf9 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 03:04:15.913055 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d89396576884f7c8901dba54f783947" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:15.906634 24622 raft_consensus.cc:2454] T 74443a0a1dc34d88a2f42e3b722db405 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 1.
I20260925 03:04:15.762581 25505 raft_consensus.cc:504] T 5278a60f97da4e10aecd50a8d9a9a43c P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:15.908756 24542 raft_consensus.cc:2454] T b030d84c670b48d5bff337ccc0c77626 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 1.
I20260925 03:04:15.909613 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f22ddd7d1c444dc58f45a1fb0b3e02bc" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:15.910750 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c23337a7cb74184a0e57b7095cb6a11" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:15.762452 25542 raft_consensus.cc:504] T 100c113a9fd84cc8adf2ec7b33ea0093 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:15.747633 25531 raft_consensus.cc:504] T 7993f9c3e2a44f408bbf5d1e395e2306 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:15.913430 25539 raft_consensus.cc:482] T e6b6ec010caf4689b8c90449764f5cf9 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:15.913915 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06f4a7eaf6534e4986c9fd35911038e4" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:15.914047 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b8e0d260a884ecdb20f79de46c36b19" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:15.905349 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5332a5d329493991dff994e70ea92c" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:15.914740 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b5df303cd6a4d59aca5c18bbbff24ed" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:15.927700 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25102ba08e4e49388f6176badcd10a25" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:15.961932 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ac285175464fbda67211d9574db415" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:15.928424 24467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ac285175464fbda67211d9574db415" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:15.894596 25536 raft_consensus.cc:504] T c85f5d1d3f004cf5817bb594fd8e6469 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:15.909924 25489 raft_consensus.cc:482] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:15.928678 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f68804eb5a244d87bb395d50683c6012" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:15.911602 25501 leader_election.cc:288] T a4abe0ea757e4886be4945c5001e529a P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:15.908937 25517 raft_consensus.cc:482] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:15.965137 24473 raft_consensus.cc:2454] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 1.
I20260925 03:04:15.932310 24465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5332a5d329493991dff994e70ea92c" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:15.963088 24543 raft_consensus.cc:2454] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 1.
I20260925 03:04:15.963814 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71f461a6e2ff434993b78a2d077f0abd" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:15.914892 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48ca6ccd98194cf2af7b5278d15bf4ea" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:15.909396 25490 raft_consensus.cc:482] T b4ebd2b043024a548005c8b6b00b0449 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:15.933321 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abd5b8dc1b7e49e3aa9af900d73ff0e7" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:15.968570 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e725b82985746b39c0cc8f671c95353" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:15.968752 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74443a0a1dc34d88a2f42e3b722db405" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:15.932245 25540 leader_election.cc:288] T 48ca6ccd98194cf2af7b5278d15bf4ea P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:15.970824 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21c43af5e6844fdaacc8c625fa01fd8d" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:15.971160 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "293ef69caf514ac98510a68495f54eda" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
W20260925 03:04:15.966166 24402 debug-util.cc:398] Leaking SignalData structure 0x7b08002e8120 after lost signal to thread 25223
W20260925 03:04:15.973488 24402 debug-util.cc:398] Leaking SignalData structure 0x7b08002e8060 after lost signal to thread 25375
I20260925 03:04:15.966816 25489 raft_consensus.cc:3037] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:15.969290 25517 raft_consensus.cc:3037] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:15.973667 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:15.975557 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abd5b8dc1b7e49e3aa9af900d73ff0e7" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:15.977264 24467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43366a0a2f9d4344bf4e37ffd290ef8a" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:15.931743 25539 raft_consensus.cc:3037] T e6b6ec010caf4689b8c90449764f5cf9 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Advancing to term 2
W20260925 03:04:15.973745 24402 debug-util.cc:398] Leaking SignalData structure 0x7b080026f0c0 after lost signal to thread 25412
I20260925 03:04:15.965633 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b627ecb4a5384ef392aa695e06599c39" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:15.976366 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7691c6b475428c82d6181d693d26f3" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:15.976126 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60d45b7d467d4e1ab92367f6d7d52b15" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:15.977329 24542 raft_consensus.cc:2454] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 1.
I20260925 03:04:15.931844 25165 leader_election.cc:302] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5; no voters:
I20260925 03:04:15.978956 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6b6ec010caf4689b8c90449764f5cf9" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:15.979274 25536 leader_election.cc:288] T c85f5d1d3f004cf5817bb594fd8e6469 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
W20260925 03:04:15.985620 24402 debug-util.cc:398] Leaking SignalData structure 0x7b080021f380 after lost signal to thread 25422
I20260925 03:04:15.975098 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e5819c15eac4544a57e5ee6c9d4f117" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:15.972239 25490 raft_consensus.cc:3037] T b4ebd2b043024a548005c8b6b00b0449 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:16.050532 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4abe0ea757e4886be4945c5001e529a" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:16.039116 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7aa47c48404465b94a36fd819fe3c7a" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:16.039569 24615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcf6e6a422d4d198d4fbb0f7d3d845d" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:16.036237 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b030d84c670b48d5bff337ccc0c77626" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:16.069728 25197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15861cd28e354598a3c6bfc4337c34bb" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:15.979847 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5f9b503b0ab43999d582015a367441e" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:16.034029 25530 leader_election.cc:288] T 100c113a9fd84cc8adf2ec7b33ea0093 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:16.042572 25482 raft_consensus.cc:1050] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5: attempting to promote NON_VOTER 1bc1f2e62ab44af995165bd5116a9d43 to VOTER
I20260925 03:04:16.044112 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e068a0f80941dc9138cee063f53bab" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:16.041971 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4ebd2b043024a548005c8b6b00b0449" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:16.082208 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be4e0aa984f4f2b81a58e382e434ad5" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:16.082397 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e725b82985746b39c0cc8f671c95353" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:16.083496 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f22ddd7d1c444dc58f45a1fb0b3e02bc" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:16.057572 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9c33fd100ff44c1a3247319a324a044" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:16.084466 24436 leader_election.cc:302] T 37a157996e944db2906110d87ecb0c54 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:16.034205 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:16.085129 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f68804eb5a244d87bb395d50683c6012" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:15.986027 25531 leader_election.cc:288] T 7993f9c3e2a44f408bbf5d1e395e2306 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:15.996011 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5332a5d329493991dff994e70ea92c" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:16.087131 24436 leader_election.cc:302] T 442d4a29b1d743599f4b6b4871e26668 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:16.084520 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:16.087564 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06f4a7eaf6534e4986c9fd35911038e4" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:16.046489 24614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c81a1b3b616492ab6c964f45af36077" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:15.997200 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d89396576884f7c8901dba54f783947" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:15.994134 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceab6b0fccd54e909a64f66fe81a9a1f" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:16.039207 24616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceab6b0fccd54e909a64f66fe81a9a1f" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:16.045069 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1e85248d3b04b979dd080caeeb02fdb" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:16.044605 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e54ca1757ee4e42956a9cebbeca4f62" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:16.052721 24618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71f461a6e2ff434993b78a2d077f0abd" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:16.090152 25477 raft_consensus.cc:2781] T 37a157996e944db2906110d87ecb0c54 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 03:04:16.052038 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ac466793d094325ace224708b2158ed" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:16.090567 25477 raft_consensus.cc:482] T 37a157996e944db2906110d87ecb0c54 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:16.088065 24544 raft_consensus.cc:2454] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 1.
I20260925 03:04:16.091939 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5aec38e66a947c681e6c2a72b05e6ca" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:16.092895 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b627ecb4a5384ef392aa695e06599c39" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:16.095082 24615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e8af2641e124f9589c5ccfdac785f63" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:16.113440 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ac466793d094325ace224708b2158ed" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:16.114334 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60d45b7d467d4e1ab92367f6d7d52b15" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:16.116663 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7993f9c3e2a44f408bbf5d1e395e2306" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:16.117460 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e068a0f80941dc9138cee063f53bab" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:16.083909 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5f9b503b0ab43999d582015a367441e" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:16.113196 25524 raft_consensus.cc:2781] T 442d4a29b1d743599f4b6b4871e26668 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 03:04:16.035988 25542 leader_election.cc:288] T 100c113a9fd84cc8adf2ec7b33ea0093 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:16.113511 24467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1414dcfe53f64a78ae413531eb8ec117" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:16.122258 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442d4a29b1d743599f4b6b4871e26668" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:16.116340 24436 leader_election.cc:302] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:16.123900 25488 raft_consensus.cc:2781] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 03:04:16.124253 25488 raft_consensus.cc:482] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:16.124577 25488 raft_consensus.cc:3037] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:16.118078 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5278a60f97da4e10aecd50a8d9a9a43c" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:16.101133 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6b6ec010caf4689b8c90449764f5cf9" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:16.126883 24436 leader_election.cc:302] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:16.064209 25505 leader_election.cc:288] T 5278a60f97da4e10aecd50a8d9a9a43c P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:16.115554 25197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ac466793d094325ace224708b2158ed" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:16.115161 25162 leader_election.cc:302] T a4abe0ea757e4886be4945c5001e529a P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:16.115506 25544 raft_consensus.cc:2781] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 03:04:15.994697 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "293ef69caf514ac98510a68495f54eda" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:16.128201 25197 raft_consensus.cc:2454] T 8ac466793d094325ace224708b2158ed P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 1.
I20260925 03:04:16.128499 25544 raft_consensus.cc:482] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:16.128957 25544 raft_consensus.cc:3037] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:16.129364 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4abe0ea757e4886be4945c5001e529a" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:16.130193 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "550118bc1a0641ce9803bdbf43f2069f" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:16.129330 25489 raft_consensus.cc:504] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:16.120440 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4ebd2b043024a548005c8b6b00b0449" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
W20260925 03:04:16.087900 24402 debug-util.cc:398] Leaking SignalData structure 0x7b08002f5e00 after lost signal to thread 25442
W20260925 03:04:16.132189 24402 debug-util.cc:398] Leaking SignalData structure 0x7b08002a9c20 after lost signal to thread 25445
I20260925 03:04:16.120568 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b8e0d260a884ecdb20f79de46c36b19" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:16.132936 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c88cbd39e01f44628a094af516be2833" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:16.133100 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b030d84c670b48d5bff337ccc0c77626" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:16.133219 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f22ddd7d1c444dc58f45a1fb0b3e02bc" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:16.133481 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15861cd28e354598a3c6bfc4337c34bb" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
W20260925 03:04:16.095614 25165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 166230us
I20260925 03:04:16.118513 25524 raft_consensus.cc:482] T 442d4a29b1d743599f4b6b4871e26668 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:16.178382 25524 raft_consensus.cc:3037] T 442d4a29b1d743599f4b6b4871e26668 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:16.179464 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d89396576884f7c8901dba54f783947" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:16.179750 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:16.181149 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:16.182556 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bb46bce3564841b7674e6a693bd2e6" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:16.184078 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c23337a7cb74184a0e57b7095cb6a11" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:16.083354 25482 consensus_queue.cc:237] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:16.130589 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7691c6b475428c82d6181d693d26f3" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:16.091028 25477 raft_consensus.cc:3037] T 37a157996e944db2906110d87ecb0c54 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:16.185508 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7aa47c48404465b94a36fd819fe3c7a" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:16.185900 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b030d84c670b48d5bff337ccc0c77626" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:16.186966 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e8af2641e124f9589c5ccfdac785f63" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:16.188864 24586 leader_election.cc:302] T 11bb46bce3564841b7674e6a693bd2e6 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
W20260925 03:04:16.132448 24402 debug-util.cc:398] Leaking SignalData structure 0x7b0800300ae0 after lost signal to thread 25448
I20260925 03:04:16.134166 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:16.132387 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e5819c15eac4544a57e5ee6c9d4f117" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
W20260925 03:04:16.189479 24402 debug-util.cc:398] Leaking SignalData structure 0x7b08002a3660 after lost signal to thread 25449
W20260925 03:04:16.189814 24402 debug-util.cc:398] Leaking SignalData structure 0x7b08003008a0 after lost signal to thread 25453
W20260925 03:04:16.190104 24402 debug-util.cc:398] Leaking SignalData structure 0x7b0800272ec0 after lost signal to thread 25454
I20260925 03:04:16.190235 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48ca6ccd98194cf2af7b5278d15bf4ea" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:16.191457 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abd5b8dc1b7e49e3aa9af900d73ff0e7" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
W20260925 03:04:16.190356 24402 debug-util.cc:398] Leaking SignalData structure 0x7b08003209a0 after lost signal to thread 25457
I20260925 03:04:16.191960 24586 leader_election.cc:302] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:16.079669 25539 raft_consensus.cc:504] T e6b6ec010caf4689b8c90449764f5cf9 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:16.121070 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c88cbd39e01f44628a094af516be2833" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:16.193001 25531 raft_consensus.cc:2781] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 03:04:16.193377 25531 raft_consensus.cc:482] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting leader election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:16.193707 25531 raft_consensus.cc:3037] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:16.194051 25539 leader_election.cc:288] T e6b6ec010caf4689b8c90449764f5cf9 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:16.192381 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:16.195591 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceab6b0fccd54e909a64f66fe81a9a1f" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:16.196262 24473 raft_consensus.cc:1226] T e6b6ec010caf4689b8c90449764f5cf9 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 1. Current term is 2. Ops: []
I20260925 03:04:16.135464 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74443a0a1dc34d88a2f42e3b722db405" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:16.196570 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7aa47c48404465b94a36fd819fe3c7a" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:16.197767 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60d45b7d467d4e1ab92367f6d7d52b15" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:16.189283 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "420163a6cdce412ebc21c25fe3835638" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:16.198565 25165 leader_election.cc:302] T b4ebd2b043024a548005c8b6b00b0449 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:16.154210 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25102ba08e4e49388f6176badcd10a25" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:16.120182 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e068a0f80941dc9138cee063f53bab" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:16.162377 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d89396576884f7c8901dba54f783947" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:16.134033 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcf6e6a422d4d198d4fbb0f7d3d845d" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:16.164616 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b8e0d260a884ecdb20f79de46c36b19" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:16.165195 24541 raft_consensus.cc:2454] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 1.
I20260925 03:04:16.166285 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e92f377a79354a6ebb4eec67303c13e3" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:16.200807 25163 leader_election.cc:302] T 04e068a0f80941dc9138cee063f53bab P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:16.163514 24467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06f4a7eaf6534e4986c9fd35911038e4" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:16.161224 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "420163a6cdce412ebc21c25fe3835638" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:16.090621 24618 raft_consensus.cc:2454] T 71f461a6e2ff434993b78a2d077f0abd P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 1.
I20260925 03:04:16.202653 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25102ba08e4e49388f6176badcd10a25" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:16.169123 25489 leader_election.cc:288] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 election: Requested vote from peers eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:16.170833 25162 leader_election.cc:302] T b1e85248d3b04b979dd080caeeb02fdb P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:16.204178 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f1fac028a144058b92331c0d9b44ebf" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:16.171676 25517 raft_consensus.cc:504] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:16.170965 25488 raft_consensus.cc:504] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:16.172853 24614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21c43af5e6844fdaacc8c625fa01fd8d" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:16.206090 25517 leader_election.cc:288] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 2 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:16.172576 25545 raft_consensus.cc:2726] T a4abe0ea757e4886be4945c5001e529a P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 03:04:16.173863 25487 raft_consensus.cc:2781] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 03:04:16.174160 24615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43366a0a2f9d4344bf4e37ffd290ef8a" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:16.175101 24436 leader_election.cc:302] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:16.175734 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e5819c15eac4544a57e5ee6c9d4f117" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:16.177608 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1414dcfe53f64a78ae413531eb8ec117" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:15.996843 24469 raft_consensus.cc:2454] T b627ecb4a5384ef392aa695e06599c39 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 1.
I20260925 03:04:16.189076 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ac285175464fbda67211d9574db415" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:16.189669 25505 raft_consensus.cc:2781] T 11bb46bce3564841b7674e6a693bd2e6 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 03:04:16.197021 25524 raft_consensus.cc:504] T 442d4a29b1d743599f4b6b4871e26668 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:16.166165 25490 raft_consensus.cc:504] T b4ebd2b043024a548005c8b6b00b0449 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:16.201632 24468 raft_consensus.cc:2454] T 7b8e0d260a884ecdb20f79de46c36b19 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 1.
I20260925 03:04:16.169035 24465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcf6e6a422d4d198d4fbb0f7d3d845d" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:16.173445 24539 raft_consensus.cc:2454] T f22ddd7d1c444dc58f45a1fb0b3e02bc P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 1.
I20260925 03:04:16.209547 24436 leader_election.cc:302] T 7b8e0d260a884ecdb20f79de46c36b19 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:16.219043 24436 leader_election.cc:302] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:16.220088 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7691c6b475428c82d6181d693d26f3" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:16.210424 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ac466793d094325ace224708b2158ed" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:16.213842 24586 leader_election.cc:302] T c85f5d1d3f004cf5817bb594fd8e6469 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:16.214818 25545 raft_consensus.cc:2726] T 04e068a0f80941dc9138cee063f53bab P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 03:04:16.215991 25517 raft_consensus.cc:2781] T b4ebd2b043024a548005c8b6b00b0449 P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 03:04:16.224099 24586 leader_election.cc:302] T 367277daf8d04778b3fa4ed94d71580d P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:16.213675 25505 raft_consensus.cc:482] T 11bb46bce3564841b7674e6a693bd2e6 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting leader election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:16.225849 25489 raft_consensus.cc:2781] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 03:04:16.227557 24511 leader_election.cc:302] T 8ac466793d094325ace224708b2158ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:04:16.215929 25539 raft_consensus.cc:2781] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 03:04:16.228374 24472 raft_consensus.cc:1226] T b4ebd2b043024a548005c8b6b00b0449 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 1. Current term is 2. Ops: []
I20260925 03:04:16.228645 25545 raft_consensus.cc:2726] T b1e85248d3b04b979dd080caeeb02fdb P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 03:04:16.223085 25513 raft_consensus.cc:2781] T 7b8e0d260a884ecdb20f79de46c36b19 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 03:04:16.229265 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e8af2641e124f9589c5ccfdac785f63" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:16.230062 25487 raft_consensus.cc:482] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:16.230044 25505 raft_consensus.cc:3037] T 11bb46bce3564841b7674e6a693bd2e6 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:16.228737 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9357d144aa49789faaf31fe5e6bf89" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:16.235725 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5278a60f97da4e10aecd50a8d9a9a43c" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:16.244727 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bb46bce3564841b7674e6a693bd2e6" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:16.246218 24586 leader_election.cc:302] T 420163a6cdce412ebc21c25fe3835638 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:16.227494 25507 raft_consensus.cc:2781] T c85f5d1d3f004cf5817bb594fd8e6469 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 03:04:16.230922 25488 leader_election.cc:288] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:16.226789 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7993f9c3e2a44f408bbf5d1e395e2306" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:16.251638 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "100c113a9fd84cc8adf2ec7b33ea0093" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:16.231446 25517 raft_consensus.cc:482] T b4ebd2b043024a548005c8b6b00b0449 P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:16.251852 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "100c113a9fd84cc8adf2ec7b33ea0093" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:16.255972 24586 leader_election.cc:302] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:16.241640 25540 raft_consensus.cc:2781] T 8ac466793d094325ace224708b2158ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 03:04:16.258001 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "550118bc1a0641ce9803bdbf43f2069f" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:16.258416 24586 leader_election.cc:302] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:16.242748 25530 raft_consensus.cc:2781] T 367277daf8d04778b3fa4ed94d71580d P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 03:04:16.259298 24436 leader_election.cc:302] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:16.234004 25539 raft_consensus.cc:482] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:16.245132 25513 raft_consensus.cc:482] T 7b8e0d260a884ecdb20f79de46c36b19 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:16.233254 25489 raft_consensus.cc:482] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:16.249794 25487 raft_consensus.cc:3037] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:16.269124 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c23337a7cb74184a0e57b7095cb6a11" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:16.275835 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc2df40db08649c9a75e90000b878559" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:16.249939 25507 raft_consensus.cc:482] T c85f5d1d3f004cf5817bb594fd8e6469 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting leader election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:16.257697 25517 raft_consensus.cc:3037] T b4ebd2b043024a548005c8b6b00b0449 P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Advancing to term 2
W20260925 03:04:16.243570 25162 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 72816us
W20260925 03:04:16.265795 25165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 67323us
I20260925 03:04:16.255038 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367277daf8d04778b3fa4ed94d71580d" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:16.277002 24586 leader_election.cc:302] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:16.288777 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74443a0a1dc34d88a2f42e3b722db405" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:16.252300 24469 raft_consensus.cc:2454] T 9e8af2641e124f9589c5ccfdac785f63 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 1.
I20260925 03:04:16.275719 25537 raft_consensus.cc:2781] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 03:04:16.295001 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48ca6ccd98194cf2af7b5278d15bf4ea" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:16.266791 25501 raft_consensus.cc:2781] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 03:04:16.260912 25535 raft_consensus.cc:2781] T 420163a6cdce412ebc21c25fe3835638 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 03:04:16.275802 25513 raft_consensus.cc:3037] T 7b8e0d260a884ecdb20f79de46c36b19 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:16.301306 25535 raft_consensus.cc:482] T 420163a6cdce412ebc21c25fe3835638 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting leader election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:16.301683 25535 raft_consensus.cc:3037] T 420163a6cdce412ebc21c25fe3835638 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:16.267195 25540 raft_consensus.cc:482] T 8ac466793d094325ace224708b2158ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:16.268834 25539 raft_consensus.cc:3037] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:16.293331 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7aa47c48404465b94a36fd819fe3c7a" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
W20260925 03:04:16.278993 25163 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 78314.8us
I20260925 03:04:16.268739 25530 raft_consensus.cc:482] T 367277daf8d04778b3fa4ed94d71580d P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting leader election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:16.311671 25530 raft_consensus.cc:3037] T 367277daf8d04778b3fa4ed94d71580d P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:16.277938 25489 raft_consensus.cc:3037] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:16.298425 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "100c113a9fd84cc8adf2ec7b33ea0093" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:16.300542 24465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e725b82985746b39c0cc8f671c95353" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:16.289220 25507 raft_consensus.cc:3037] T c85f5d1d3f004cf5817bb594fd8e6469 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:16.314661 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c85f5d1d3f004cf5817bb594fd8e6469" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:16.291505 25552 raft_consensus.cc:2781] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 03:04:16.302306 25533 raft_consensus.cc:2781] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 03:04:16.297166 25501 raft_consensus.cc:482] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting leader election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:16.312986 25540 raft_consensus.cc:3037] T 8ac466793d094325ace224708b2158ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:16.294646 25537 raft_consensus.cc:482] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting leader election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:16.325031 25524 leader_election.cc:288] T 442d4a29b1d743599f4b6b4871e26668 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:16.326308 24436 leader_election.cc:302] T 06f4a7eaf6534e4986c9fd35911038e4 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:16.326857 24586 leader_election.cc:302] T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:16.327710 24436 leader_election.cc:302] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:16.322789 25490 leader_election.cc:288] T b4ebd2b043024a548005c8b6b00b0449 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:16.329993 24465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "420163a6cdce412ebc21c25fe3835638" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:16.330479 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ac285175464fbda67211d9574db415" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:16.331794 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc2df40db08649c9a75e90000b878559" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:16.332811 25488 raft_consensus.cc:2781] T 06f4a7eaf6534e4986c9fd35911038e4 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 03:04:16.332893 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367277daf8d04778b3fa4ed94d71580d" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:16.333242 25488 raft_consensus.cc:482] T 06f4a7eaf6534e4986c9fd35911038e4 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:16.333617 25488 raft_consensus.cc:3037] T 06f4a7eaf6534e4986c9fd35911038e4 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:16.296277 25531 raft_consensus.cc:504] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:16.318717 25552 raft_consensus.cc:482] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:16.341471 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9357d144aa49789faaf31fe5e6bf89" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:16.349234 24465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f1fac028a144058b92331c0d9b44ebf" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:16.349476 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48ca6ccd98194cf2af7b5278d15bf4ea" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:16.318599 25165 leader_election.cc:302] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:16.351590 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25102ba08e4e49388f6176badcd10a25" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:16.330370 25544 raft_consensus.cc:504] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:16.353381 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15861cd28e354598a3c6bfc4337c34bb" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:16.352944 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "550118bc1a0641ce9803bdbf43f2069f" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:16.357062 24436 leader_election.cc:302] T 4e5332a5d329493991dff994e70ea92c P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:16.318321 25533 raft_consensus.cc:482] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting leader election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:16.323511 25501 raft_consensus.cc:3037] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:16.328787 25537 raft_consensus.cc:3037] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:16.363464 24436 leader_election.cc:302] T b627ecb4a5384ef392aa695e06599c39 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:16.357628 25554 raft_consensus.cc:2781] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 03:04:16.368767 24436 leader_election.cc:302] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:16.318882 25162 leader_election.cc:302] T 8ac466793d094325ace224708b2158ed P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:16.383040 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "100c113a9fd84cc8adf2ec7b33ea0093" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:16.384791 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c85f5d1d3f004cf5817bb594fd8e6469" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:16.385429 24436 leader_election.cc:302] T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:16.360257 25552 raft_consensus.cc:3037] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:16.367987 25533 raft_consensus.cc:3037] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:16.368604 25482 raft_consensus.cc:2781] T 4e5332a5d329493991dff994e70ea92c P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 03:04:16.381590 25548 raft_consensus.cc:2781] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 03:04:16.381634 25490 raft_consensus.cc:2781] T b627ecb4a5384ef392aa695e06599c39 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 03:04:16.402290 25524 raft_consensus.cc:2781] T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 03:04:16.388065 25553 raft_consensus.cc:2781] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 03:04:16.404908 24514 leader_election.cc:302] T 0be4e0aa984f4f2b81a58e382e434ad5 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
W20260925 03:04:16.409577 24511 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 90755.5us
I20260925 03:04:16.386373 25554 raft_consensus.cc:482] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:16.364873 25517 raft_consensus.cc:504] T b4ebd2b043024a548005c8b6b00b0449 P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:16.397188 25544 leader_election.cc:288] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 2 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
W20260925 03:04:16.389070 25165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 74795.1us
I20260925 03:04:16.397559 25482 raft_consensus.cc:482] T 4e5332a5d329493991dff994e70ea92c P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:16.409528 25548 raft_consensus.cc:482] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:16.409353 25490 raft_consensus.cc:482] T b627ecb4a5384ef392aa695e06599c39 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:16.425549 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7993f9c3e2a44f408bbf5d1e395e2306" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:16.415035 25524 raft_consensus.cc:482] T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:16.414072 25553 raft_consensus.cc:482] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:16.399108 25547 raft_consensus.cc:2726] T 8ac466793d094325ace224708b2158ed P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260925 03:04:16.416918 25162 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 107001us
I20260925 03:04:16.439193 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c88cbd39e01f44628a094af516be2833" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:16.419807 25554 raft_consensus.cc:3037] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:16.433938 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5278a60f97da4e10aecd50a8d9a9a43c" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:16.413578 25163 leader_election.cc:302] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:04:16.447167 25539 raft_consensus.cc:504] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:16.448614 25531 leader_election.cc:288] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:16.433214 25482 raft_consensus.cc:3037] T 4e5332a5d329493991dff994e70ea92c P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Advancing to term 2
W20260925 03:04:16.460462 25163 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 62504us
I20260925 03:04:16.421938 25505 raft_consensus.cc:504] T 11bb46bce3564841b7674e6a693bd2e6 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:16.434499 25490 raft_consensus.cc:3037] T b627ecb4a5384ef392aa695e06599c39 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:16.433106 25548 raft_consensus.cc:3037] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:16.434381 25536 raft_consensus.cc:2726] T 0be4e0aa984f4f2b81a58e382e434ad5 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 03:04:16.436873 25524 raft_consensus.cc:3037] T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:16.439754 25553 raft_consensus.cc:3037] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Advancing to term 2
W20260925 03:04:16.465610 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 78596.7us
I20260925 03:04:16.462818 25544 raft_consensus.cc:2781] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 03:04:16.478302 24514 leader_election.cc:302] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
W20260925 03:04:16.479112 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 62004.4us
I20260925 03:04:16.480480 24514 leader_election.cc:302] T edcf6e6a422d4d198d4fbb0f7d3d845d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
W20260925 03:04:16.481750 24436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 96377.2us
I20260925 03:04:16.468850 25531 raft_consensus.cc:2781] T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 03:04:16.465451 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5f9b503b0ab43999d582015a367441e" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:16.464798 25533 raft_consensus.cc:504] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:16.480044 25544 raft_consensus.cc:482] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:16.482172 25508 raft_consensus.cc:2726] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260925 03:04:16.484359 25516 raft_consensus.cc:2726] T edcf6e6a422d4d198d4fbb0f7d3d845d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 03:04:16.473544 25162 leader_election.cc:302] T 71f461a6e2ff434993b78a2d077f0abd P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, eeab93abfdb141a895002ec9f312296c; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:16.489559 25531 raft_consensus.cc:482] T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting leader election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:16.491245 25544 raft_consensus.cc:3037] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:16.485555 25477 raft_consensus.cc:504] T 37a157996e944db2906110d87ecb0c54 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:04:16.575187 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 71614.1us
I20260925 03:04:16.575987 24514 leader_election.cc:302] T f22ddd7d1c444dc58f45a1fb0b3e02bc P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:16.506618 25533 leader_election.cc:288] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:16.539112 25531 raft_consensus.cc:3037] T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Advancing to term 2
W20260925 03:04:16.584487 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 257712us
I20260925 03:04:16.578459 25547 raft_consensus.cc:2781] T 71f461a6e2ff434993b78a2d077f0abd P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 03:04:16.588467 24514 leader_election.cc:302] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:04:16.578473 25517 leader_election.cc:288] T b4ebd2b043024a548005c8b6b00b0449 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 2 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:16.591737 25542 raft_consensus.cc:2726] T f22ddd7d1c444dc58f45a1fb0b3e02bc P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 03:04:16.592132 25565 raft_consensus.cc:482] T 1414dcfe53f64a78ae413531eb8ec117 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
W20260925 03:04:16.595906 24436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 110904us
I20260925 03:04:16.598513 25488 raft_consensus.cc:504] T 06f4a7eaf6534e4986c9fd35911038e4 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:16.599349 25477 leader_election.cc:288] T 37a157996e944db2906110d87ecb0c54 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:16.595721 25565 raft_consensus.cc:504] T 1414dcfe53f64a78ae413531eb8ec117 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:16.597675 24436 leader_election.cc:302] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
W20260925 03:04:16.585083 25162 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 123769us
I20260925 03:04:16.600472 25488 leader_election.cc:288] T 06f4a7eaf6534e4986c9fd35911038e4 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:16.606166 24436 leader_election.cc:302] T c88cbd39e01f44628a094af516be2833 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:16.590663 25547 raft_consensus.cc:482] T 71f461a6e2ff434993b78a2d077f0abd P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:16.607605 25505 leader_election.cc:288] T 11bb46bce3564841b7674e6a693bd2e6 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:16.620678 25489 raft_consensus.cc:504] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:16.603696 25542 raft_consensus.cc:2781] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260925 03:04:16.626915 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 147205us
W20260925 03:04:16.627341 24511 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 203507us
I20260925 03:04:16.630074 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c81a1b3b616492ab6c964f45af36077" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:16.632180 24511 leader_election.cc:302] T 3d89396576884f7c8901dba54f783947 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:16.608553 25539 leader_election.cc:288] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:16.634203 24511 leader_election.cc:302] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:16.635061 24511 leader_election.cc:302] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:16.636384 24511 leader_election.cc:302] T b030d84c670b48d5bff337ccc0c77626 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:16.613271 25477 raft_consensus.cc:2781] T c88cbd39e01f44628a094af516be2833 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 03:04:16.638748 24511 leader_election.cc:302] T 21c43af5e6844fdaacc8c625fa01fd8d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:16.611451 25547 raft_consensus.cc:3037] T 71f461a6e2ff434993b78a2d077f0abd P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:16.611264 25564 raft_consensus.cc:2781] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 03:04:16.640580 24511 leader_election.cc:302] T 43366a0a2f9d4344bf4e37ffd290ef8a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:16.630890 25501 raft_consensus.cc:504] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:16.678303 24511 leader_election.cc:302] T 8e5819c15eac4544a57e5ee6c9d4f117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:16.676893 25564 raft_consensus.cc:482] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:16.628573 25542 raft_consensus.cc:482] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:16.637445 25508 raft_consensus.cc:2726] T b030d84c670b48d5bff337ccc0c77626 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 03:04:16.622102 25490 raft_consensus.cc:504] T b627ecb4a5384ef392aa695e06599c39 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:16.620954 25562 consensus_queue.cc:1046] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.057s
I20260925 03:04:16.676126 24514 leader_election.cc:302] T 11bb46bce3564841b7674e6a693bd2e6 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:16.637523 25521 raft_consensus.cc:2726] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 03:04:16.625654 25487 raft_consensus.cc:504] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:16.639938 25477 raft_consensus.cc:482] T c88cbd39e01f44628a094af516be2833 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:16.691509 25477 raft_consensus.cc:3037] T c88cbd39e01f44628a094af516be2833 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:16.691809 25530 raft_consensus.cc:504] T 367277daf8d04778b3fa4ed94d71580d P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } }
I20260925 03:04:16.694604 25540 raft_consensus.cc:504] T 8ac466793d094325ace224708b2158ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:16.696329 25562 raft_consensus.cc:3032] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 LEADER]: Stepping down as leader of term 1
I20260925 03:04:16.673974 25536 raft_consensus.cc:2726] T 21c43af5e6844fdaacc8c625fa01fd8d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 03:04:16.696846 25540 leader_election.cc:288] T 8ac466793d094325ace224708b2158ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 election: Requested vote from peers eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:16.679898 24511 leader_election.cc:302] T 1414dcfe53f64a78ae413531eb8ec117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:16.678289 25555 raft_consensus.cc:2726] T 43366a0a2f9d4344bf4e37ffd290ef8a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 03:04:16.637831 25516 raft_consensus.cc:2726] T 3d89396576884f7c8901dba54f783947 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 03:04:16.638381 25559 raft_consensus.cc:2726] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 03:04:16.699314 24514 leader_election.cc:302] T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:16.697520 25562 raft_consensus.cc:729] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:04:16.680680 25489 leader_election.cc:288] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:16.683276 25564 raft_consensus.cc:3037] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:16.679545 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c81a1b3b616492ab6c964f45af36077" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:16.685920 25542 raft_consensus.cc:3037] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:16.708423 24514 leader_election.cc:302] T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
W20260925 03:04:16.711030 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 114372us
I20260925 03:04:16.698252 25508 raft_consensus.cc:2726] T 11bb46bce3564841b7674e6a693bd2e6 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 03:04:16.703048 25565 leader_election.cc:288] T 1414dcfe53f64a78ae413531eb8ec117 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:16.716709 24514 leader_election.cc:302] T a7aa47c48404465b94a36fd819fe3c7a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:16.718135 25536 raft_consensus.cc:2781] T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 03:04:16.704193 25497 raft_consensus.cc:2726] T 1414dcfe53f64a78ae413531eb8ec117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 03:04:16.702066 25530 leader_election.cc:288] T 367277daf8d04778b3fa4ed94d71580d P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
W20260925 03:04:16.720880 25165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 284046us
I20260925 03:04:16.722359 25552 raft_consensus.cc:504] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:16.722970 25501 leader_election.cc:288] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:16.724218 25199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ac466793d094325ace224708b2158ed" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43"
I20260925 03:04:16.725353 25544 raft_consensus.cc:504] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:16.716005 25531 raft_consensus.cc:504] T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } }
W20260925 03:04:16.726300 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 199853us
I20260925 03:04:16.712360 25537 raft_consensus.cc:504] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:16.724257 25557 raft_consensus.cc:482] T b1e85248d3b04b979dd080caeeb02fdb P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:16.770334 25557 raft_consensus.cc:504] T b1e85248d3b04b979dd080caeeb02fdb P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:16.772259 25557 leader_election.cc:288] T b1e85248d3b04b979dd080caeeb02fdb P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:16.730999 24543 raft_consensus.cc:1226] T 8ac466793d094325ace224708b2158ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 1. Current term is 2. Ops: []
I20260925 03:04:16.773869 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35992316cb8d4c028333388eff02ecde" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:16.775074 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6b6ec010caf4689b8c90449764f5cf9" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:16.736788 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1414dcfe53f64a78ae413531eb8ec117" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:16.776257 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7993f9c3e2a44f408bbf5d1e395e2306" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:16.777213 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442d4a29b1d743599f4b6b4871e26668" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:16.737531 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c23337a7cb74184a0e57b7095cb6a11" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:16.727494 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f1fac028a144058b92331c0d9b44ebf" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:16.681317 25560 raft_consensus.cc:2726] T 8e5819c15eac4544a57e5ee6c9d4f117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 03:04:16.724121 25554 raft_consensus.cc:504] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:16.711587 25566 raft_consensus.cc:2726] T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 03:04:16.753561 25516 raft_consensus.cc:2726] T a7aa47c48404465b94a36fd819fe3c7a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 03:04:16.728524 25536 raft_consensus.cc:482] T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:16.755795 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e8af2641e124f9589c5ccfdac785f63" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:16.756870 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5278a60f97da4e10aecd50a8d9a9a43c" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:16.769837 25537 leader_election.cc:288] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:16.778108 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4ebd2b043024a548005c8b6b00b0449" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:16.714815 25521 consensus_queue.cc:1046] T b4ebd2b043024a548005c8b6b00b0449 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.063s
W20260925 03:04:16.778963 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 146342us
I20260925 03:04:16.751791 25199 raft_consensus.cc:3037] T 8ac466793d094325ace224708b2158ed P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:16.796824 24588 leader_election.cc:302] T 37a157996e944db2906110d87ecb0c54 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:16.797505 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35992316cb8d4c028333388eff02ecde" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:16.799468 24622 raft_consensus.cc:1226] T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 1. Current term is 2. Ops: []
I20260925 03:04:16.780977 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bb46bce3564841b7674e6a693bd2e6" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:16.800609 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442d4a29b1d743599f4b6b4871e26668" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:16.727766 25533 raft_consensus.cc:482] T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:16.801791 25533 raft_consensus.cc:504] T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:16.803663 25533 leader_election.cc:288] T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:16.713402 25535 raft_consensus.cc:504] T 420163a6cdce412ebc21c25fe3835638 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:16.804093 24619 raft_consensus.cc:1226] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 1. Current term is 2. Ops: []
I20260925 03:04:16.804952 24620 raft_consensus.cc:1226] T 420163a6cdce412ebc21c25fe3835638 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 1. Current term is 2. Ops: []
I20260925 03:04:16.805430 25535 leader_election.cc:288] T 420163a6cdce412ebc21c25fe3835638 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:16.753679 24514 leader_election.cc:302] T 420163a6cdce412ebc21c25fe3835638 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
W20260925 03:04:16.806906 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 53310.6us
I20260925 03:04:16.807329 24619 raft_consensus.cc:1226] T 367277daf8d04778b3fa4ed94d71580d P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 1. Current term is 2. Ops: []
I20260925 03:04:16.807576 24514 leader_election.cc:302] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:16.807874 25559 raft_consensus.cc:2726] T 420163a6cdce412ebc21c25fe3835638 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 03:04:16.808364 25560 raft_consensus.cc:2726] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 03:04:16.780766 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ac285175464fbda67211d9574db415" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:16.808898 24615 raft_consensus.cc:1226] T 11bb46bce3564841b7674e6a693bd2e6 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 1. Current term is 2. Ops: []
I20260925 03:04:16.809333 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "420163a6cdce412ebc21c25fe3835638" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:16.790277 24437 leader_election.cc:302] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:04:16.810212 24615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37a157996e944db2906110d87ecb0c54" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:16.804158 25199 raft_consensus.cc:2454] T 8ac466793d094325ace224708b2158ed P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 2.
I20260925 03:04:16.808792 24514 leader_election.cc:302] T cc2df40db08649c9a75e90000b878559 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:16.812884 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be4e0aa984f4f2b81a58e382e434ad5" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:16.812991 24614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abd5b8dc1b7e49e3aa9af900d73ff0e7" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:16.809026 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367277daf8d04778b3fa4ed94d71580d" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:16.815763 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06f4a7eaf6534e4986c9fd35911038e4" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:16.788760 25536 raft_consensus.cc:3037] T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:16.817842 24437 leader_election.cc:302] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:04:16.822103 25554 leader_election.cc:288] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:16.792096 25557 consensus_queue.cc:1046] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.026s
I20260925 03:04:16.825770 25566 raft_consensus.cc:2726] T cc2df40db08649c9a75e90000b878559 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 03:04:16.820458 24588 leader_election.cc:302] T 21c43af5e6844fdaacc8c625fa01fd8d P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:16.826833 25539 raft_consensus.cc:2781] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 03:04:16.827204 25539 raft_consensus.cc:482] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:16.827610 25490 leader_election.cc:288] T b627ecb4a5384ef392aa695e06599c39 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:16.828754 25487 leader_election.cc:288] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:16.829551 25537 raft_consensus.cc:2726] T 21c43af5e6844fdaacc8c625fa01fd8d P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 03:04:16.829867 25531 leader_election.cc:288] T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:16.827553 24588 leader_election.cc:302] T 43366a0a2f9d4344bf4e37ffd290ef8a P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:16.831414 25564 raft_consensus.cc:504] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:16.832309 25505 raft_consensus.cc:2726] T 43366a0a2f9d4344bf4e37ffd290ef8a P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 03:04:16.833495 25524 raft_consensus.cc:504] T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:16.834801 24547 raft_consensus.cc:1261] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260925 03:04:16.786724 25165 leader_election.cc:302] T e6b6ec010caf4689b8c90449764f5cf9 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:16.836150 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b030d84c670b48d5bff337ccc0c77626" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
W20260925 03:04:16.836347 25165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 57672.9us
I20260925 03:04:16.837102 25165 leader_election.cc:302] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:16.833531 24588 leader_election.cc:302] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:16.835072 24437 leader_election.cc:302] T 8e5819c15eac4544a57e5ee6c9d4f117 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:16.839342 24588 leader_election.cc:302] T 550118bc1a0641ce9803bdbf43f2069f P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:16.840014 24511 leader_election.cc:302] T 8ac466793d094325ace224708b2158ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:04:16.820618 25544 leader_election.cc:288] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 2 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:16.821199 25574 raft_consensus.cc:2726] T 37a157996e944db2906110d87ecb0c54 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 03:04:16.826399 25557 raft_consensus.cc:3032] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [term 1 LEADER]: Stepping down as leader of term 1
I20260925 03:04:16.844364 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:16.845424 24514 leader_election.cc:302] T 48ca6ccd98194cf2af7b5278d15bf4ea P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:16.817606 25567 raft_consensus.cc:2781] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 03:04:16.827600 25539 raft_consensus.cc:3037] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:16.846652 24514 leader_election.cc:302] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:16.851477 24514 leader_election.cc:302] T 25102ba08e4e49388f6176badcd10a25 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:16.852092 25567 raft_consensus.cc:482] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:16.854163 25567 raft_consensus.cc:3037] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:16.844179 25517 raft_consensus.cc:2781] T e6b6ec010caf4689b8c90449764f5cf9 P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 03:04:16.853658 24437 leader_election.cc:302] T 25102ba08e4e49388f6176badcd10a25 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:16.779963 25562 consensus_queue.cc:260] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:16.844738 25571 raft_consensus.cc:2726] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 03:04:16.844354 25521 raft_consensus.cc:2781] T 8ac466793d094325ace224708b2158ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 03:04:16.872786 25542 raft_consensus.cc:504] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:16.873216 25521 raft_consensus.cc:686] T 8ac466793d094325ace224708b2158ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 LEADER]: Becoming Leader. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:04:16.874094 25521 consensus_queue.cc:237] T 8ac466793d094325ace224708b2158ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:16.874773 25548 raft_consensus.cc:504] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:16.875859 25198 raft_consensus.cc:1226] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 1. Current term is 2. Ops: []
I20260925 03:04:16.876350 25553 raft_consensus.cc:504] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:16.876777 25548 leader_election.cc:288] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 2 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:16.878258 25553 leader_election.cc:288] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:16.868611 25477 raft_consensus.cc:504] T c88cbd39e01f44628a094af516be2833 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:16.859277 24615 raft_consensus.cc:1226] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 1. Current term is 2. Ops: []
I20260925 03:04:16.856776 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9c33fd100ff44c1a3247319a324a044" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:16.848433 25559 raft_consensus.cc:2726] T 48ca6ccd98194cf2af7b5278d15bf4ea P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 03:04:16.842211 25487 raft_consensus.cc:2726] T 8e5819c15eac4544a57e5ee6c9d4f117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 03:04:16.865710 25481 raft_consensus.cc:3032] T b4ebd2b043024a548005c8b6b00b0449 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 LEADER]: Stepping down as leader of term 1
I20260925 03:04:16.866027 25540 raft_consensus.cc:2726] T 25102ba08e4e49388f6176badcd10a25 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 03:04:16.866662 24614 raft_consensus.cc:1226] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 1. Current term is 2. Ops: []
I20260925 03:04:16.855062 25560 raft_consensus.cc:2726] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 03:04:16.864135 24588 leader_election.cc:302] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:16.843498 25557 raft_consensus.cc:729] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:16.868517 25490 raft_consensus.cc:2726] T 25102ba08e4e49388f6176badcd10a25 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 03:04:16.846625 25544 raft_consensus.cc:2726] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 03:04:16.870028 25199 raft_consensus.cc:1226] T b4ebd2b043024a548005c8b6b00b0449 P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 1. Current term is 2. Ops: []
I20260925 03:04:16.856263 24514 leader_election.cc:302] T ac9357d144aa49789faaf31fe5e6bf89 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:16.899632 25557 consensus_queue.cc:260] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:16.900624 25560 raft_consensus.cc:2726] T ac9357d144aa49789faaf31fe5e6bf89 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 03:04:16.902099 25557 raft_consensus.cc:3037] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Advancing to term 2
W20260925 03:04:16.888068 25507 consensus_meta.cc:287] T c85f5d1d3f004cf5817bb594fd8e6469 P eeab93abfdb141a895002ec9f312296c: Time spent flushing consensus metadata: real 0.568s user 0.003s sys 0.006s
I20260925 03:04:16.893134 24437 leader_election.cc:302] T e92f377a79354a6ebb4eec67303c13e3 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:16.904649 25501 raft_consensus.cc:2726] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 03:04:16.905238 25564 leader_election.cc:288] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:16.904649 25490 raft_consensus.cc:2726] T e92f377a79354a6ebb4eec67303c13e3 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 03:04:16.903224 25507 raft_consensus.cc:504] T c85f5d1d3f004cf5817bb594fd8e6469 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:16.906396 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60d45b7d467d4e1ab92367f6d7d52b15" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:16.907644 24622 raft_consensus.cc:1226] T c85f5d1d3f004cf5817bb594fd8e6469 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 1. Current term is 2. Ops: []
I20260925 03:04:16.909157 25507 leader_election.cc:288] T c85f5d1d3f004cf5817bb594fd8e6469 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:16.910558 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c85f5d1d3f004cf5817bb594fd8e6469" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:16.890584 25547 raft_consensus.cc:504] T 71f461a6e2ff434993b78a2d077f0abd P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
W20260925 03:04:16.884907 25516 consensus_queue.cc:1159] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "1bc1f2e62ab44af995165bd5116a9d43" responder_term: 1 status { last_received { term: 1 index: 4 } last_committed_idx: 4 last_received_current_leader { term: 1 index: 4 } }
I20260925 03:04:16.916103 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b627ecb4a5384ef392aa695e06599c39" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
W20260925 03:04:16.916795 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 60583.5us
I20260925 03:04:16.917420 24514 leader_election.cc:302] T 550118bc1a0641ce9803bdbf43f2069f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:16.848320 25505 raft_consensus.cc:2726] T 550118bc1a0641ce9803bdbf43f2069f P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 03:04:16.920257 25516 raft_consensus.cc:2726] T 550118bc1a0641ce9803bdbf43f2069f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 03:04:16.881057 25482 raft_consensus.cc:504] T 4e5332a5d329493991dff994e70ea92c P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:16.854694 25517 raft_consensus.cc:482] T e6b6ec010caf4689b8c90449764f5cf9 P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:16.921396 25517 raft_consensus.cc:3037] T e6b6ec010caf4689b8c90449764f5cf9 P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:16.922977 25482 leader_election.cc:288] T 4e5332a5d329493991dff994e70ea92c P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:16.912017 25555 consensus_queue.cc:1046] T b4ebd2b043024a548005c8b6b00b0449 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.007s
I20260925 03:04:16.926379 25562 raft_consensus.cc:3037] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Advancing to term 2
W20260925 03:04:16.853327 25513 consensus_meta.cc:287] T 7b8e0d260a884ecdb20f79de46c36b19 P 869b0a8575b54d08ae2817fad5dd42d5: Time spent flushing consensus metadata: real 0.549s user 0.005s sys 0.012s
W20260925 03:04:16.898334 25165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 61323.4us
I20260925 03:04:16.894968 25481 raft_consensus.cc:729] T b4ebd2b043024a548005c8b6b00b0449 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:04:16.867779 25524 leader_election.cc:288] T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:16.857980 25552 leader_election.cc:288] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:16.874732 25542 leader_election.cc:288] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:16.914348 24588 leader_election.cc:302] T 48ca6ccd98194cf2af7b5278d15bf4ea P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:16.931995 25481 consensus_queue.cc:260] T b4ebd2b043024a548005c8b6b00b0449 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:16.933297 24588 leader_election.cc:302] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:16.933570 25481 raft_consensus.cc:3037] T b4ebd2b043024a548005c8b6b00b0449 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:16.934860 24588 leader_election.cc:302] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:16.935833 24588 leader_election.cc:302] T a7aa47c48404465b94a36fd819fe3c7a P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:16.936362 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b5df303cd6a4d59aca5c18bbbff24ed" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:16.914494 24437 leader_election.cc:302] T f22ddd7d1c444dc58f45a1fb0b3e02bc P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:04:16.888041 25477 leader_election.cc:288] T c88cbd39e01f44628a094af516be2833 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:16.891227 25508 consensus_queue.cc:1046] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260925 03:04:16.937197 25572 raft_consensus.cc:2726] T a7aa47c48404465b94a36fd819fe3c7a P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 03:04:16.937382 25501 raft_consensus.cc:2726] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260925 03:04:16.940554 25552 raft_consensus.cc:2781] T f22ddd7d1c444dc58f45a1fb0b3e02bc P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 03:04:16.941354 25552 raft_consensus.cc:482] T f22ddd7d1c444dc58f45a1fb0b3e02bc P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:16.941625 25552 raft_consensus.cc:3037] T f22ddd7d1c444dc58f45a1fb0b3e02bc P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:16.941937 25505 raft_consensus.cc:2726] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 03:04:16.931541 25539 raft_consensus.cc:504] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } }
I20260925 03:04:16.938112 24437 leader_election.cc:302] T 7993f9c3e2a44f408bbf5d1e395e2306 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:16.951118 24437 leader_election.cc:302] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:16.938519 24588 leader_election.cc:302] T 3d89396576884f7c8901dba54f783947 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:16.928794 25513 raft_consensus.cc:504] T 7b8e0d260a884ecdb20f79de46c36b19 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:16.947053 25560 raft_consensus.cc:3032] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 LEADER]: Stepping down as leader of term 1
I20260925 03:04:16.948396 25198 raft_consensus.cc:1226] T 8ac466793d094325ace224708b2158ed P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 1. Current term is 2. Ops: []
I20260925 03:04:16.949510 25199 raft_consensus.cc:1226] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 1. Current term is 2. Ops: []
I20260925 03:04:16.952250 24437 leader_election.cc:302] T 9e8af2641e124f9589c5ccfdac785f63 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:16.952296 24588 leader_election.cc:302] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:16.953022 25524 raft_consensus.cc:2726] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 03:04:16.953100 25195 raft_consensus.cc:1226] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 1. Current term is 2. Ops: []
I20260925 03:04:16.937656 25507 raft_consensus.cc:2726] T 48ca6ccd98194cf2af7b5278d15bf4ea P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 03:04:16.953886 25560 raft_consensus.cc:729] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:04:16.966933 25490 consensus_queue.cc:1035] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260925 03:04:16.967559 25539 leader_election.cc:288] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:16.955783 25196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5aec38e66a947c681e6c2a72b05e6ca" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:16.968495 25547 leader_election.cc:288] T 71f461a6e2ff434993b78a2d077f0abd P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 2 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:16.969120 25196 raft_consensus.cc:2454] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 1.
I20260925 03:04:16.970021 25567 raft_consensus.cc:504] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:16.934477 25536 raft_consensus.cc:504] T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:16.953861 25477 raft_consensus.cc:2726] T 7993f9c3e2a44f408bbf5d1e395e2306 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 03:04:16.959367 25513 leader_election.cc:288] T 7b8e0d260a884ecdb20f79de46c36b19 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:16.959818 25482 raft_consensus.cc:2726] T 9e8af2641e124f9589c5ccfdac785f63 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 03:04:16.957975 25505 raft_consensus.cc:2726] T 3d89396576884f7c8901dba54f783947 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 03:04:16.962457 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4ebd2b043024a548005c8b6b00b0449" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:16.962651 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15861cd28e354598a3c6bfc4337c34bb" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:16.963253 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71f461a6e2ff434993b78a2d077f0abd" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:16.964584 24514 leader_election.cc:302] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:04:16.965385 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "293ef69caf514ac98510a68495f54eda" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:16.972357 25501 raft_consensus.cc:2726] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 03:04:16.974483 25560 consensus_queue.cc:260] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:16.978688 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abd5b8dc1b7e49e3aa9af900d73ff0e7" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:16.981980 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9c33fd100ff44c1a3247319a324a044" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:16.979161 24588 leader_election.cc:302] T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:16.982298 25560 raft_consensus.cc:3037] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:16.979357 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37a157996e944db2906110d87ecb0c54" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:16.980934 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06f4a7eaf6534e4986c9fd35911038e4" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:16.983868 25501 raft_consensus.cc:2726] T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 03:04:16.985541 25196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71f461a6e2ff434993b78a2d077f0abd" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:16.989209 24620 raft_consensus.cc:1226] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 1. Current term is 2. Ops: []
I20260925 03:04:16.990942 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74443a0a1dc34d88a2f42e3b722db405" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:16.989476 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60d45b7d467d4e1ab92367f6d7d52b15" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:16.991905 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e725b82985746b39c0cc8f671c95353" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:16.992061 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5332a5d329493991dff994e70ea92c" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:16.992290 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b8e0d260a884ecdb20f79de46c36b19" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:16.989876 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b627ecb4a5384ef392aa695e06599c39" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:16.993347 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5f9b503b0ab43999d582015a367441e" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:16.993495 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c88cbd39e01f44628a094af516be2833" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:16.994199 24545 raft_consensus.cc:1226] T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 1. Current term is 2. Ops: []
I20260925 03:04:16.990094 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b5df303cd6a4d59aca5c18bbbff24ed" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:16.995114 25536 leader_election.cc:288] T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:16.990530 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7691c6b475428c82d6181d693d26f3" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:16.993844 24588 leader_election.cc:302] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:16.997221 24588 leader_election.cc:302] T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:17.000600 25505 raft_consensus.cc:2781] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 03:04:17.001204 24469 raft_consensus.cc:1226] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 1. Current term is 2. Ops: []
I20260925 03:04:17.001451 25199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4abe0ea757e4886be4945c5001e529a" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:17.001659 24465 raft_consensus.cc:1226] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 1. Current term is 2. Ops: []
I20260925 03:04:17.001662 25501 raft_consensus.cc:2781] T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 03:04:17.004261 25501 raft_consensus.cc:482] T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting leader election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:17.005904 24588 leader_election.cc:302] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:17.006090 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15861cd28e354598a3c6bfc4337c34bb" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:17.006969 24588 leader_election.cc:302] T edcf6e6a422d4d198d4fbb0f7d3d845d P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:17.002524 24471 raft_consensus.cc:1226] T 7b8e0d260a884ecdb20f79de46c36b19 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 1. Current term is 2. Ops: []
I20260925 03:04:17.009543 24467 raft_consensus.cc:1226] T c88cbd39e01f44628a094af516be2833 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 1. Current term is 2. Ops: []
I20260925 03:04:17.009960 24588 leader_election.cc:302] T 100c113a9fd84cc8adf2ec7b33ea0093 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:17.004995 25501 raft_consensus.cc:3037] T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:17.002595 25507 consensus_queue.cc:1046] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20260925 03:04:17.010423 24436 leader_election.cc:302] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:04:17.010639 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4ebd2b043024a548005c8b6b00b0449" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:17.011089 25572 raft_consensus.cc:2726] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 03:04:17.011869 24469 raft_consensus.cc:2379] T b4ebd2b043024a548005c8b6b00b0449 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1bc1f2e62ab44af995165bd5116a9d43 in current term 2: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:04:17.012014 25507 raft_consensus.cc:3032] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c [term 1 LEADER]: Stepping down as leader of term 1
I20260925 03:04:17.012398 25507 raft_consensus.cc:729] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:17.001266 25505 raft_consensus.cc:482] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting leader election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:17.012612 25513 raft_consensus.cc:2781] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 03:04:17.011147 24470 raft_consensus.cc:1226] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 1. Current term is 2. Ops: []
I20260925 03:04:17.005473 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c81a1b3b616492ab6c964f45af36077" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:17.011340 25530 raft_consensus.cc:2726] T edcf6e6a422d4d198d4fbb0f7d3d845d P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 03:04:17.013337 25513 raft_consensus.cc:482] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:17.012880 24469 raft_consensus.cc:1226] T 4e5332a5d329493991dff994e70ea92c P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 1. Current term is 2. Ops: []
I20260925 03:04:17.013983 25513 raft_consensus.cc:3037] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:17.014284 24467 raft_consensus.cc:1226] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 1. Current term is 2. Ops: []
I20260925 03:04:17.012570 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "293ef69caf514ac98510a68495f54eda" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:17.016500 25530 raft_consensus.cc:2726] T 100c113a9fd84cc8adf2ec7b33ea0093 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 03:04:17.011371 24465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e54ca1757ee4e42956a9cebbeca4f62" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:17.013154 24473 raft_consensus.cc:1226] T b627ecb4a5384ef392aa695e06599c39 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 1. Current term is 2. Ops: []
I20260925 03:04:17.018530 24469 raft_consensus.cc:1226] T 37a157996e944db2906110d87ecb0c54 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 1. Current term is 2. Ops: []
I20260925 03:04:17.018643 24436 leader_election.cc:302] T 293ef69caf514ac98510a68495f54eda P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: 1bc1f2e62ab44af995165bd5116a9d43, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:17.013338 25505 raft_consensus.cc:3037] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:17.018994 25507 consensus_queue.cc:260] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:17.020291 25507 raft_consensus.cc:3037] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:17.020560 24469 raft_consensus.cc:1226] T 06f4a7eaf6534e4986c9fd35911038e4 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 1. Current term is 2. Ops: []
I20260925 03:04:17.021171 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f68804eb5a244d87bb395d50683c6012" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:17.021581 25482 raft_consensus.cc:2726] T 293ef69caf514ac98510a68495f54eda P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 03:04:17.023430 24466 raft_consensus.cc:1226] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 1. Current term is 2. Ops: []
I20260925 03:04:17.024732 24469 raft_consensus.cc:1226] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 1. Current term is 2. Ops: []
I20260925 03:04:17.026429 24547 raft_consensus.cc:1226] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 1. Current term is 2. Ops: []
I20260925 03:04:16.983538 25481 raft_consensus.cc:2726] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 03:04:17.027853 25530 consensus_queue.cc:1046] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260925 03:04:17.030061 25530 raft_consensus.cc:3032] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c [term 1 LEADER]: Stepping down as leader of term 1
I20260925 03:04:17.030591 25530 raft_consensus.cc:729] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:17.031402 25530 consensus_queue.cc:260] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:17.030061 25567 leader_election.cc:288] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:17.037979 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e8af2641e124f9589c5ccfdac785f63" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:17.038429 24465 raft_consensus.cc:1226] T 442d4a29b1d743599f4b6b4871e26668 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 1. Current term is 2. Ops: []
I20260925 03:04:17.039187 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:17.039453 24469 raft_consensus.cc:1226] T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 1. Current term is 2. Ops: []
I20260925 03:04:17.040329 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7691c6b475428c82d6181d693d26f3" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:17.040551 24614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74443a0a1dc34d88a2f42e3b722db405" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:17.041630 24616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e725b82985746b39c0cc8f671c95353" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:17.044313 24615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c88cbd39e01f44628a094af516be2833" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:17.044446 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:17.045544 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5f9b503b0ab43999d582015a367441e" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:17.046286 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b8e0d260a884ecdb20f79de46c36b19" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:17.046890 24614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5332a5d329493991dff994e70ea92c" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:16.977160 25552 raft_consensus.cc:504] T f22ddd7d1c444dc58f45a1fb0b3e02bc P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:17.048203 24616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f68804eb5a244d87bb395d50683c6012" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:17.051292 25552 leader_election.cc:288] T f22ddd7d1c444dc58f45a1fb0b3e02bc P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:17.036130 25517 raft_consensus.cc:504] T e6b6ec010caf4689b8c90449764f5cf9 P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:17.052837 24436 leader_election.cc:302] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:17.036484 25530 raft_consensus.cc:3037] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:17.056201 25482 raft_consensus.cc:2726] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260925 03:04:17.057497 25482 consensus_queue.cc:1046] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260925 03:04:17.037130 24588 leader_election.cc:302] T 74443a0a1dc34d88a2f42e3b722db405 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:17.060653 24436 leader_election.cc:302] T 442d4a29b1d743599f4b6b4871e26668 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:17.062247 25197 raft_consensus.cc:1261] T 7e49248643df4131a1c42e74244e05c2 P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 LEARNER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260925 03:04:17.063117 25199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c81a1b3b616492ab6c964f45af36077" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43"
I20260925 03:04:17.063807 25199 raft_consensus.cc:2379] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 in current term 2: Already voted for candidate 1bc1f2e62ab44af995165bd5116a9d43 in this term.
I20260925 03:04:17.063556 24540 raft_consensus.cc:2454] T b4ebd2b043024a548005c8b6b00b0449 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 2.
I20260925 03:04:17.064512 25477 raft_consensus.cc:3032] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 LEADER]: Stepping down as leader of term 1
I20260925 03:04:17.065084 25477 raft_consensus.cc:729] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:04:17.066606 25477 consensus_queue.cc:260] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:17.072470 25198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6b6ec010caf4689b8c90449764f5cf9" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43"
I20260925 03:04:17.073858 25477 raft_consensus.cc:3037] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:17.074355 25562 consensus_queue.cc:1046] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260925 03:04:17.075240 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6b6ec010caf4689b8c90449764f5cf9" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:17.065234 25557 raft_consensus.cc:2726] T 74443a0a1dc34d88a2f42e3b722db405 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 03:04:17.072494 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f68804eb5a244d87bb395d50683c6012" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:17.078383 24588 leader_election.cc:302] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:17.080090 24588 leader_election.cc:302] T 7993f9c3e2a44f408bbf5d1e395e2306 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:17.081811 24588 leader_election.cc:302] T c88cbd39e01f44628a094af516be2833 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:17.082610 24588 leader_election.cc:302] T 5278a60f97da4e10aecd50a8d9a9a43c P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:17.081079 25563 raft_consensus.cc:2726] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 03:04:17.081727 25501 raft_consensus.cc:504] T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:17.081209 25497 consensus_queue.cc:1046] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.004s
I20260925 03:04:17.085541 25574 raft_consensus.cc:2726] T c88cbd39e01f44628a094af516be2833 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 03:04:17.086189 25497 raft_consensus.cc:3032] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 LEADER]: Stepping down as leader of term 1
I20260925 03:04:17.086584 25497 raft_consensus.cc:729] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:04:17.081487 25521 consensus_queue.cc:1046] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.006s
I20260925 03:04:17.087751 25521 raft_consensus.cc:3032] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 LEADER]: Stepping down as leader of term 1
I20260925 03:04:17.087571 25497 consensus_queue.cc:260] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:17.081233 25571 raft_consensus.cc:2726] T 7993f9c3e2a44f408bbf5d1e395e2306 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 03:04:17.088411 25513 raft_consensus.cc:504] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:17.089849 25198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4ebd2b043024a548005c8b6b00b0449" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43"
I20260925 03:04:17.090348 25513 leader_election.cc:288] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 election: Requested vote from peers eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:17.077793 25517 leader_election.cc:288] T e6b6ec010caf4689b8c90449764f5cf9 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 2 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:17.091006 25501 leader_election.cc:288] T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:17.091787 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b627ecb4a5384ef392aa695e06599c39" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:17.093452 25536 raft_consensus.cc:3032] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 LEADER]: Stepping down as leader of term 1
I20260925 03:04:17.093818 25536 raft_consensus.cc:729] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:04:17.094645 25536 consensus_queue.cc:260] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:17.095844 25536 raft_consensus.cc:3037] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:17.088109 25521 raft_consensus.cc:729] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:04:17.097910 24543 raft_consensus.cc:2454] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 2.
I20260925 03:04:17.096128 25485 consensus_queue.cc:1046] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260925 03:04:17.100680 25542 consensus_queue.cc:1046] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260925 03:04:17.101866 25542 consensus_queue.cc:1046] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260925 03:04:17.102588 25566 consensus_queue.cc:1046] T 37a157996e944db2906110d87ecb0c54 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260925 03:04:17.084931 25163 leader_election.cc:302] T b4ebd2b043024a548005c8b6b00b0449 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:04:17.051411 25195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e068a0f80941dc9138cee063f53bab" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:17.102735 25542 consensus_queue.cc:1046] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.004s
I20260925 03:04:17.105897 25540 consensus_queue.cc:1046] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260925 03:04:17.110862 24588 leader_election.cc:302] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:17.079190 25567 raft_consensus.cc:2726] T 442d4a29b1d743599f4b6b4871e26668 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260925 03:04:17.100881 25503 consensus_queue.cc:1046] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.014s
I20260925 03:04:17.115571 25521 consensus_queue.cc:260] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:17.117061 25521 raft_consensus.cc:3037] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:17.123670 25571 raft_consensus.cc:2726] T 5278a60f97da4e10aecd50a8d9a9a43c P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 03:04:17.124825 25566 raft_consensus.cc:3032] T 37a157996e944db2906110d87ecb0c54 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 LEADER]: Stepping down as leader of term 1
I20260925 03:04:17.125203 25566 raft_consensus.cc:729] T 37a157996e944db2906110d87ecb0c54 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:04:17.125581 25533 raft_consensus.cc:2726] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 03:04:17.110553 25555 consensus_queue.cc:1046] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.016s
I20260925 03:04:17.126950 25555 raft_consensus.cc:3032] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 LEADER]: Stepping down as leader of term 1
I20260925 03:04:17.125348 25531 raft_consensus.cc:1050] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c: attempting to promote NON_VOTER 1bc1f2e62ab44af995165bd5116a9d43 to VOTER
I20260925 03:04:17.126809 25566 consensus_queue.cc:260] T 37a157996e944db2906110d87ecb0c54 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:17.128324 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f1fac028a144058b92331c0d9b44ebf" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:17.128623 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ac285175464fbda67211d9574db415" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:17.147383 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e8af2641e124f9589c5ccfdac785f63" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:17.127954 24511 leader_election.cc:302] T 100c113a9fd84cc8adf2ec7b33ea0093 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:17.130427 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367277daf8d04778b3fa4ed94d71580d" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:17.130671 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c23337a7cb74184a0e57b7095cb6a11" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:17.125229 25539 consensus_queue.cc:1046] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.004s
I20260925 03:04:17.131748 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1e85248d3b04b979dd080caeeb02fdb" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:17.132396 24467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "420163a6cdce412ebc21c25fe3835638" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:17.133011 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be4e0aa984f4f2b81a58e382e434ad5" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:17.103510 25497 raft_consensus.cc:3037] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:17.110690 25559 raft_consensus.cc:3032] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 LEADER]: Stepping down as leader of term 1
I20260925 03:04:17.133379 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b030d84c670b48d5bff337ccc0c77626" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:17.134008 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1414dcfe53f64a78ae413531eb8ec117" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:17.139093 25542 raft_consensus.cc:3032] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 LEADER]: Stepping down as leader of term 1
I20260925 03:04:17.129068 24465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bb46bce3564841b7674e6a693bd2e6" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:17.127339 25555 raft_consensus.cc:729] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:04:17.146538 25572 raft_consensus.cc:1050] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c: attempting to promote NON_VOTER 1bc1f2e62ab44af995165bd5116a9d43 to VOTER
I20260925 03:04:17.146574 24617 raft_consensus.cc:1226] T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 1. Current term is 2. Ops: []
I20260925 03:04:17.154618 25559 raft_consensus.cc:729] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:04:17.152974 24472 raft_consensus.cc:1226] T f22ddd7d1c444dc58f45a1fb0b3e02bc P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 1. Current term is 2. Ops: []
I20260925 03:04:17.154479 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c85f5d1d3f004cf5817bb594fd8e6469" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:17.164044 25564 consensus_queue.cc:1046] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: INVALID_TERM, Last received: 1.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.007s
I20260925 03:04:17.171365 25564 raft_consensus.cc:3032] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 LEADER]: Stepping down as leader of term 1
I20260925 03:04:17.171689 25564 raft_consensus.cc:729] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:04:17.169400 25198 raft_consensus.cc:1226] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 1. Current term is 2. Ops: []
I20260925 03:04:17.152153 24588 leader_election.cc:302] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:17.154232 24614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ac466793d094325ace224708b2158ed" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:17.158084 24548 raft_consensus.cc:2454] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 2.
I20260925 03:04:17.174073 24588 leader_election.cc:302] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:17.174918 24588 leader_election.cc:302] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:17.175573 24436 leader_election.cc:302] T 37a157996e944db2906110d87ecb0c54 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:17.151645 24437 leader_election.cc:302] T e6b6ec010caf4689b8c90449764f5cf9 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:04:17.110004 25562 raft_consensus.cc:3032] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 LEADER]: Stepping down as leader of term 1
I20260925 03:04:17.176265 25563 raft_consensus.cc:2726] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260925 03:04:17.176204 25574 raft_consensus.cc:2726] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260925 03:04:17.158860 25542 raft_consensus.cc:729] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:04:17.100303 25481 consensus_queue.cc:1046] T c88cbd39e01f44628a094af516be2833 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.020s
I20260925 03:04:17.164162 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b627ecb4a5384ef392aa695e06599c39" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:17.177330 25552 consensus_queue.cc:1046] T 11bb46bce3564841b7674e6a693bd2e6 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260925 03:04:17.119585 25505 raft_consensus.cc:504] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:17.165295 25503 raft_consensus.cc:3032] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 LEADER]: Stepping down as leader of term 1
I20260925 03:04:17.176186 25562 raft_consensus.cc:729] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:04:17.154129 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:17.169528 25540 raft_consensus.cc:2726] T 100c113a9fd84cc8adf2ec7b33ea0093 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 03:04:17.170382 25559 consensus_queue.cc:260] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:17.135416 25517 raft_consensus.cc:2781] T b4ebd2b043024a548005c8b6b00b0449 P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 03:04:17.166430 25539 raft_consensus.cc:3032] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 LEADER]: Stepping down as leader of term 1
I20260925 03:04:17.162906 24371 catalog_manager.cc:5495] T 8ac466793d094325ace224708b2158ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 reported cstate change: term changed from 1 to 2, leader changed from eeab93abfdb141a895002ec9f312296c (127.22.193.3) to a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2). New cstate: current_term: 2 leader_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260925 03:04:17.169349 24511 leader_election.cc:302] T c85f5d1d3f004cf5817bb594fd8e6469 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:17.152576 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f68804eb5a244d87bb395d50683c6012" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:17.163950 25571 consensus_queue.cc:1046] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.021s
I20260925 03:04:17.180678 25503 raft_consensus.cc:729] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:04:17.179690 24436 leader_election.cc:302] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:17.190989 24622 raft_consensus.cc:2454] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 2.
I20260925 03:04:17.179557 24588 leader_election.cc:302] T 420163a6cdce412ebc21c25fe3835638 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:17.178562 24585 leader_election.cc:302] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:04:17.185302 24614 raft_consensus.cc:2454] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 2.
I20260925 03:04:17.133730 25531 consensus_queue.cc:237] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } attrs { replace: true } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:17.191011 24436 leader_election.cc:302] T 06f4a7eaf6534e4986c9fd35911038e4 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:17.180509 25482 raft_consensus.cc:2726] T 37a157996e944db2906110d87ecb0c54 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260925 03:04:17.186415 24511 leader_election.cc:302] T 367277daf8d04778b3fa4ed94d71580d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:17.181659 25574 raft_consensus.cc:2726] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260925 03:04:17.181171 25567 raft_consensus.cc:3032] T 11bb46bce3564841b7674e6a693bd2e6 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 LEADER]: Stepping down as leader of term 1
I20260925 03:04:17.179894 25481 raft_consensus.cc:3032] T c88cbd39e01f44628a094af516be2833 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 LEADER]: Stepping down as leader of term 1
I20260925 03:04:17.201184 24588 leader_election.cc:302] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:17.180114 25529 raft_consensus.cc:3032] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 LEADER]: Stepping down as leader of term 1
I20260925 03:04:17.202829 25558 raft_consensus.cc:2726] T 06f4a7eaf6534e4986c9fd35911038e4 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260925 03:04:17.203054 24466 raft_consensus.cc:2396] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c for term 2 because replica has last-logged OpId of term: 1 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 2.
I20260925 03:04:17.202772 25163 leader_election.cc:302] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:04:17.206969 25508 raft_consensus.cc:3032] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 LEADER]: Stepping down as leader of term 1
I20260925 03:04:17.213429 25508 raft_consensus.cc:729] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:04:17.213517 25547 raft_consensus.cc:2781] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 03:04:17.214054 25547 raft_consensus.cc:686] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 LEADER]: Becoming Leader. State: Replica: 1bc1f2e62ab44af995165bd5116a9d43, State: Running, Role: LEADER
I20260925 03:04:17.214231 25508 consensus_queue.cc:260] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:17.214900 25547 consensus_queue.cc:237] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:17.215579 25508 raft_consensus.cc:3037] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:17.184839 25490 raft_consensus.cc:2781] T e6b6ec010caf4689b8c90449764f5cf9 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 03:04:17.182010 24613 raft_consensus.cc:1226] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 1. Current term is 2. Ops: []
I20260925 03:04:17.203888 25567 raft_consensus.cc:729] T 11bb46bce3564841b7674e6a693bd2e6 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:04:17.218587 25490 raft_consensus.cc:686] T e6b6ec010caf4689b8c90449764f5cf9 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 LEADER]: Becoming Leader. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:04:17.218670 25567 consensus_queue.cc:260] T 11bb46bce3564841b7674e6a693bd2e6 P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:17.219569 25490 consensus_queue.cc:237] T e6b6ec010caf4689b8c90449764f5cf9 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:17.184798 25517 raft_consensus.cc:686] T b4ebd2b043024a548005c8b6b00b0449 P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 LEADER]: Becoming Leader. State: Replica: 1bc1f2e62ab44af995165bd5116a9d43, State: Running, Role: LEADER
I20260925 03:04:17.207283 25529 raft_consensus.cc:729] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:04:17.181559 25562 consensus_queue.cc:260] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:17.221230 25517 consensus_queue.cc:237] T b4ebd2b043024a548005c8b6b00b0449 P 1bc1f2e62ab44af995165bd5116a9d43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:17.222223 25562 raft_consensus.cc:3037] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:17.186558 25540 raft_consensus.cc:2726] T c85f5d1d3f004cf5817bb594fd8e6469 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 03:04:17.184593 25559 raft_consensus.cc:3037] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:17.203400 25487 consensus_queue.cc:1046] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20260925 03:04:17.203277 24469 raft_consensus.cc:1226] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 1. Current term is 2. Ops: []
I20260925 03:04:17.204941 25481 raft_consensus.cc:729] T c88cbd39e01f44628a094af516be2833 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:04:17.189847 25503 consensus_queue.cc:260] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:17.225461 25487 raft_consensus.cc:3032] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 LEADER]: Stepping down as leader of term 1
I20260925 03:04:17.226167 25503 raft_consensus.cc:3037] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:17.225781 25481 consensus_queue.cc:260] T c88cbd39e01f44628a094af516be2833 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:17.201939 25537 raft_consensus.cc:2726] T 420163a6cdce412ebc21c25fe3835638 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260925 03:04:17.186281 25542 consensus_queue.cc:260] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:17.202829 25535 raft_consensus.cc:2781] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 03:04:17.185128 25539 raft_consensus.cc:729] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:04:17.206669 24588 leader_election.cc:302] T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:17.195214 25552 raft_consensus.cc:2726] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260925 03:04:17.226056 25487 raft_consensus.cc:729] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:04:17.179898 25507 consensus_queue.cc:1046] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.018s
I20260925 03:04:17.189337 25564 consensus_queue.cc:260] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } }
I20260925 03:04:17.207808 24585 leader_election.cc:302] T a4abe0ea757e4886be4945c5001e529a P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 1bc1f2e62ab44af995165bd5116a9d43, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:17.209780 24436 leader_election.cc:302] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:17.200798 25496 raft_consensus.cc:2726] T 367277daf8d04778b3fa4ed94d71580d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 03:04:17.188807 25560 consensus_queue.cc:1046] T 442d4a29b1d743599f4b6b4871e26668 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.016s
I20260925 03:04:17.190086 25485 consensus_queue.cc:1046] T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.017s
I20260925 03:04:17.207082 25574 raft_consensus.cc:2726] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260925 03:04:17.220114 25567 raft_consensus.cc:3037] T 11bb46bce3564841b7674e6a693bd2e6 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:17.187306 25566 raft_consensus.cc:3037] T 37a157996e944db2906110d87ecb0c54 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:17.230665 24588 leader_election.cc:302] T 1414dcfe53f64a78ae413531eb8ec117 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:17.237053 24614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71f461a6e2ff434993b78a2d077f0abd" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:17.239637 25542 raft_consensus.cc:3037] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:17.240012 25560 raft_consensus.cc:3032] T 442d4a29b1d743599f4b6b4871e26668 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 LEADER]: Stepping down as leader of term 1
I20260925 03:04:17.240305 24585 leader_election.cc:302] T 71f461a6e2ff434993b78a2d077f0abd P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 1bc1f2e62ab44af995165bd5116a9d43, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:17.240981 25485 raft_consensus.cc:3032] T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 LEADER]: Stepping down as leader of term 1
I20260925 03:04:17.241544 24436 leader_election.cc:302] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:17.242137 25552 consensus_queue.cc:1046] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260925 03:04:17.242287 24618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e54ca1757ee4e42956a9cebbeca4f62" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:17.238576 25513 consensus_queue.cc:1046] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.005s
I20260925 03:04:17.237263 25571 raft_consensus.cc:2726] T a4abe0ea757e4886be4945c5001e529a P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 03:04:17.228928 25481 raft_consensus.cc:3037] T c88cbd39e01f44628a094af516be2833 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:17.239737 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f22ddd7d1c444dc58f45a1fb0b3e02bc" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:17.185730 25555 consensus_queue.cc:260] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:17.251406 24436 leader_election.cc:302] T b627ecb4a5384ef392aa695e06599c39 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:17.229597 25535 raft_consensus.cc:482] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting leader election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:17.232894 25537 raft_consensus.cc:2726] T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 03:04:17.254686 24588 leader_election.cc:302] T 367277daf8d04778b3fa4ed94d71580d P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:17.255620 25574 raft_consensus.cc:2726] T 1414dcfe53f64a78ae413531eb8ec117 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 03:04:17.255808 25162 leader_election.cc:302] T 71f461a6e2ff434993b78a2d077f0abd P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:17.233758 25540 consensus_queue.cc:1046] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.022s
I20260925 03:04:17.239265 25490 raft_consensus.cc:2726] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260925 03:04:17.256896 25524 consensus_queue.cc:1046] T c85f5d1d3f004cf5817bb594fd8e6469 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260925 03:04:17.279438 25513 raft_consensus.cc:3032] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 LEADER]: Stepping down as leader of term 1
I20260925 03:04:17.324944 25537 raft_consensus.cc:2726] T 367277daf8d04778b3fa4ed94d71580d P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260925 03:04:17.325230 25540 raft_consensus.cc:3032] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 LEADER]: Stepping down as leader of term 1
I20260925 03:04:17.325387 25513 raft_consensus.cc:729] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:04:17.325855 25540 raft_consensus.cc:729] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
W20260925 03:04:17.326202 25162 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 70494us
W20260925 03:04:17.317451 24511 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 118998us
I20260925 03:04:17.250767 25560 raft_consensus.cc:729] T 442d4a29b1d743599f4b6b4871e26668 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:04:17.315096 24368 catalog_manager.cc:5495] T b4ebd2b043024a548005c8b6b00b0449 P 1bc1f2e62ab44af995165bd5116a9d43 reported cstate change: term changed from 1 to 2, leader changed from a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2) to 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4). New cstate: current_term: 2 leader_uuid: "1bc1f2e62ab44af995165bd5116a9d43" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260925 03:04:17.289234 25557 raft_consensus.cc:3032] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c [term 1 LEADER]: Stepping down as leader of term 1
I20260925 03:04:17.252874 25482 raft_consensus.cc:2726] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260925 03:04:17.284451 25535 raft_consensus.cc:3037] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:17.257651 25533 raft_consensus.cc:2726] T 71f461a6e2ff434993b78a2d077f0abd P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 03:04:17.279518 25529 consensus_queue.cc:260] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:17.297961 25555 raft_consensus.cc:3037] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:17.250478 25485 raft_consensus.cc:729] T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:04:17.259110 25539 consensus_queue.cc:260] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:17.255823 25487 consensus_queue.cc:260] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:17.299264 25165 leader_election.cc:302] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
W20260925 03:04:17.317874 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 63240us
I20260925 03:04:17.294950 25198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1e85248d3b04b979dd080caeeb02fdb" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:17.316828 25501 raft_consensus.cc:3032] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [term 1 LEADER]: Stepping down as leader of term 1
I20260925 03:04:17.322521 24585 leader_election.cc:302] T 04e068a0f80941dc9138cee063f53bab P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 1bc1f2e62ab44af995165bd5116a9d43, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:17.288298 25564 raft_consensus.cc:3037] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Advancing to term 2
W20260925 03:04:17.329219 24436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 77895.2us
I20260925 03:04:17.346176 24436 leader_election.cc:302] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:17.327272 25540 consensus_queue.cc:260] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:17.331647 25488 raft_consensus.cc:2726] T b627ecb4a5384ef392aa695e06599c39 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260925 03:04:17.349412 25544 raft_consensus.cc:2726] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260925 03:04:17.349248 25485 consensus_queue.cc:260] T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:17.280296 25572 consensus_queue.cc:237] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:17.343816 24588 leader_election.cc:302] T 11bb46bce3564841b7674e6a693bd2e6 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:17.355881 24588 leader_election.cc:302] T c85f5d1d3f004cf5817bb594fd8e6469 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:17.330451 25517 raft_consensus.cc:2726] T 71f461a6e2ff434993b78a2d077f0abd P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260925 03:04:17.337544 25557 raft_consensus.cc:729] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:17.344478 25501 raft_consensus.cc:729] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:17.347306 25533 raft_consensus.cc:2726] T 04e068a0f80941dc9138cee063f53bab P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 03:04:17.326834 25513 consensus_queue.cc:260] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:17.361049 24588 leader_election.cc:302] T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:17.350656 25529 raft_consensus.cc:3037] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:17.350661 25485 raft_consensus.cc:3037] T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:17.369017 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b8e0d260a884ecdb20f79de46c36b19" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:17.358729 25574 raft_consensus.cc:2726] T 11bb46bce3564841b7674e6a693bd2e6 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260925 03:04:17.364370 25165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 65341.5us
I20260925 03:04:17.364537 25487 raft_consensus.cc:3037] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:17.361593 25537 raft_consensus.cc:2726] T c85f5d1d3f004cf5817bb594fd8e6469 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260925 03:04:17.364127 25570 raft_consensus.cc:2726] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260925 03:04:17.378605 24588 leader_election.cc:302] T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:17.365518 25505 leader_election.cc:288] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:17.359797 25530 consensus_queue.cc:1046] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.014s
I20260925 03:04:17.391583 25568 raft_consensus.cc:2726] T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260925 03:04:17.383296 25501 consensus_queue.cc:260] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:17.394501 25501 raft_consensus.cc:3037] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:17.401019 25513 raft_consensus.cc:3037] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:17.404960 25165 leader_election.cc:302] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:17.405872 25517 raft_consensus.cc:2726] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260925 03:04:17.406430 25165 leader_election.cc:302] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:17.407124 25517 raft_consensus.cc:2726] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260925 03:04:17.407991 24585 leader_election.cc:302] T b1e85248d3b04b979dd080caeeb02fdb P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:04:17.409040 25501 raft_consensus.cc:2726] T b1e85248d3b04b979dd080caeeb02fdb P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 03:04:17.412320 25535 raft_consensus.cc:504] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:17.382091 25537 raft_consensus.cc:2726] T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260925 03:04:17.415064 25535 leader_election.cc:288] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:17.400920 24547 raft_consensus.cc:2454] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 2.
I20260925 03:04:17.418393 24586 leader_election.cc:302] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:17.419291 25535 raft_consensus.cc:2781] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Leader election won for term 2
I20260925 03:04:17.420011 25535 raft_consensus.cc:686] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [term 2 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:17.421103 25535 consensus_queue.cc:237] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:17.382179 25557 consensus_queue.cc:260] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:17.423592 25557 raft_consensus.cc:3037] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:17.444667 25540 raft_consensus.cc:3037] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:17.440460 25539 raft_consensus.cc:3037] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:17.452069 24369 catalog_manager.cc:5495] T e6b6ec010caf4689b8c90449764f5cf9 P 869b0a8575b54d08ae2817fad5dd42d5 reported cstate change: term changed from 1 to 2, leader changed from a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2) to 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1). New cstate: current_term: 2 leader_uuid: "869b0a8575b54d08ae2817fad5dd42d5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260925 03:04:17.456542 25550 raft_consensus.cc:3032] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 LEADER]: Stepping down as leader of term 1
I20260925 03:04:17.456971 25550 raft_consensus.cc:729] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:04:17.428860 24616 raft_consensus.cc:2454] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 2.
W20260925 03:04:17.472568 24436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 126470us
I20260925 03:04:17.473451 25198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5aec38e66a947c681e6c2a72b05e6ca" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43"
I20260925 03:04:17.477653 25550 consensus_queue.cc:260] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:17.478685 25560 consensus_queue.cc:260] T 442d4a29b1d743599f4b6b4871e26668 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:17.479758 25550 raft_consensus.cc:3037] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:17.484762 25198 raft_consensus.cc:1261] T e92f377a79354a6ebb4eec67303c13e3 P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 LEARNER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260925 03:04:17.494400 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f22ddd7d1c444dc58f45a1fb0b3e02bc" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:17.494726 24468 raft_consensus.cc:1226] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 1. Current term is 2. Ops: []
I20260925 03:04:17.496347 25530 consensus_queue.cc:1035] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260925 03:04:17.486773 25553 raft_consensus.cc:3032] T c85f5d1d3f004cf5817bb594fd8e6469 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 LEADER]: Stepping down as leader of term 1
I20260925 03:04:17.502460 25553 raft_consensus.cc:729] T c85f5d1d3f004cf5817bb594fd8e6469 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:04:17.504144 24540 raft_consensus.cc:1261] T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260925 03:04:17.490105 25560 raft_consensus.cc:3037] T 442d4a29b1d743599f4b6b4871e26668 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:17.506220 25533 consensus_queue.cc:1035] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260925 03:04:17.493618 25505 raft_consensus.cc:3032] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c [term 1 LEADER]: Stepping down as leader of term 1
I20260925 03:04:17.509382 25505 raft_consensus.cc:729] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:17.510532 24464 raft_consensus.cc:1261] T e92f377a79354a6ebb4eec67303c13e3 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260925 03:04:17.510900 25505 consensus_queue.cc:260] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:17.503624 25553 consensus_queue.cc:260] T c85f5d1d3f004cf5817bb594fd8e6469 P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:17.512975 25553 raft_consensus.cc:3037] T c85f5d1d3f004cf5817bb594fd8e6469 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:17.518743 25552 consensus_queue.cc:1046] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260925 03:04:17.519613 24465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5aec38e66a947c681e6c2a72b05e6ca" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:17.517107 24436 leader_election.cc:302] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:17.535326 24547 raft_consensus.cc:1261] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260925 03:04:17.532213 25533 consensus_queue.cc:1035] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.005s
I20260925 03:04:17.537340 24472 raft_consensus.cc:1226] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 1. Current term is 2. Ops: []
I20260925 03:04:17.513332 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b8e0d260a884ecdb20f79de46c36b19" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:17.545344 24436 leader_election.cc:302] T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:17.547181 25195 raft_consensus.cc:1261] T 29637f5a4a504ce6ba29033ccd0cd716 P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 LEARNER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260925 03:04:17.548436 24613 raft_consensus.cc:1261] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260925 03:04:17.548637 25557 consensus_queue.cc:1046] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260925 03:04:17.557262 25557 raft_consensus.cc:3032] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [term 1 LEADER]: Stepping down as leader of term 1
I20260925 03:04:17.557586 25557 raft_consensus.cc:729] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:17.549592 24588 leader_election.cc:302] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:04:17.559824 24469 raft_consensus.cc:1226] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 1. Current term is 2. Ops: []
I20260925 03:04:17.560289 25572 raft_consensus.cc:2726] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260925 03:04:17.549932 24511 leader_election.cc:302] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:17.550974 25199 raft_consensus.cc:1226] T 71f461a6e2ff434993b78a2d077f0abd P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 1. Current term is 2. Ops: []
I20260925 03:04:17.552011 25196 raft_consensus.cc:1261] T 8ac466793d094325ace224708b2158ed P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 FOLLOWER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260925 03:04:17.564939 24511 leader_election.cc:302] T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:04:17.556200 25589 raft_consensus.cc:2726] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260925 03:04:17.566032 25540 raft_consensus.cc:2726] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260925 03:04:17.568913 24436 leader_election.cc:302] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
W20260925 03:04:17.572652 25580 log.cc:924] Time spent T b4ebd2b043024a548005c8b6b00b0449 P 1bc1f2e62ab44af995165bd5116a9d43: Append to log took a long time: real 0.207s user 0.000s sys 0.001s
I20260925 03:04:17.572922 24466 raft_consensus.cc:1226] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 1. Current term is 2. Ops: []
I20260925 03:04:17.570425 25551 raft_consensus.cc:3032] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 LEADER]: Stepping down as leader of term 1
I20260925 03:04:17.574584 25551 raft_consensus.cc:729] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:04:17.575524 25551 consensus_queue.cc:260] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:17.581463 25551 raft_consensus.cc:3037] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:17.570546 25487 raft_consensus.cc:2726] T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260925 03:04:17.558420 25557 consensus_queue.cc:260] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:17.570755 25485 raft_consensus.cc:2781] T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 03:04:17.599975 25557 raft_consensus.cc:3037] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:17.592926 24436 leader_election.cc:302] T c88cbd39e01f44628a094af516be2833 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:17.576203 25552 raft_consensus.cc:2726] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260925 03:04:17.605001 25563 consensus_queue.cc:1159] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" responder_term: 1 status { last_received { term: 1 index: 2 } last_committed_idx: 2 error { code: PRECEDING_ENTRY_DIDNT_MATCH status { code: ILLEGAL_STATE message: "Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)" } } last_received_current_leader { term: 1 index: 2 } }
W20260925 03:04:17.607695 25572 consensus_queue.cc:1159] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "1bc1f2e62ab44af995165bd5116a9d43" responder_term: 1 status { last_received { term: 1 index: 2 } last_committed_idx: 2 error { code: PRECEDING_ENTRY_DIDNT_MATCH status { code: ILLEGAL_STATE message: "Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)" } } last_received_current_leader { term: 1 index: 2 } }
I20260925 03:04:17.592238 25536 consensus_queue.cc:1046] T 0b5df303cd6a4d59aca5c18bbbff24ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.004s
I20260925 03:04:17.608273 25564 raft_consensus.cc:2726] T c88cbd39e01f44628a094af516be2833 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260925 03:04:17.609251 25516 raft_consensus.cc:3032] T 0b5df303cd6a4d59aca5c18bbbff24ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 LEADER]: Stepping down as leader of term 1
I20260925 03:04:17.609643 25516 raft_consensus.cc:729] T 0b5df303cd6a4d59aca5c18bbbff24ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:04:17.610527 25516 consensus_queue.cc:260] T 0b5df303cd6a4d59aca5c18bbbff24ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:17.612001 25516 raft_consensus.cc:3037] T 0b5df303cd6a4d59aca5c18bbbff24ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:17.614202 24436 leader_election.cc:302] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:17.588999 25529 consensus_queue.cc:1035] T 8ac466793d094325ace224708b2158ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.031s
I20260925 03:04:17.620992 24436 leader_election.cc:302] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:17.622594 24436 leader_election.cc:302] T 7b8e0d260a884ecdb20f79de46c36b19 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:17.624053 24436 leader_election.cc:302] T 4e5332a5d329493991dff994e70ea92c P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:17.624928 25564 raft_consensus.cc:2726] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260925 03:04:17.625692 25477 raft_consensus.cc:2726] T 4e5332a5d329493991dff994e70ea92c P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260925 03:04:17.625334 25487 raft_consensus.cc:2726] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260925 03:04:17.635879 25540 consensus_queue.cc:1046] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260925 03:04:17.637378 25540 raft_consensus.cc:3032] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 LEADER]: Stepping down as leader of term 1
I20260925 03:04:17.637758 25540 raft_consensus.cc:729] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:04:17.625150 25554 raft_consensus.cc:2726] T 7b8e0d260a884ecdb20f79de46c36b19 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260925 03:04:17.639480 25521 consensus_queue.cc:1035] T 8ac466793d094325ace224708b2158ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260925 03:04:17.638564 25540 consensus_queue.cc:260] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:17.651288 24540 raft_consensus.cc:1261] T b4ebd2b043024a548005c8b6b00b0449 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Refusing update from remote peer 1bc1f2e62ab44af995165bd5116a9d43: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260925 03:04:17.598106 25505 raft_consensus.cc:3037] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:17.629444 24436 leader_election.cc:302] T f22ddd7d1c444dc58f45a1fb0b3e02bc P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:17.657305 25517 consensus_queue.cc:1035] T b4ebd2b043024a548005c8b6b00b0449 P 1bc1f2e62ab44af995165bd5116a9d43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260925 03:04:17.665493 25589 raft_consensus.cc:2726] T f22ddd7d1c444dc58f45a1fb0b3e02bc P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260925 03:04:17.667039 25540 raft_consensus.cc:3037] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:17.625186 25485 raft_consensus.cc:686] T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 LEADER]: Becoming Leader. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:04:17.671392 24436 leader_election.cc:302] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:17.692453 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e54ca1757ee4e42956a9cebbeca4f62" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:17.701581 25199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e54ca1757ee4e42956a9cebbeca4f62" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43"
I20260925 03:04:17.702488 25199 raft_consensus.cc:2379] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 in current term 2: Already voted for candidate 1bc1f2e62ab44af995165bd5116a9d43 in this term.
I20260925 03:04:17.690528 25589 raft_consensus.cc:2781] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 03:04:17.731272 24621 raft_consensus.cc:1226] T c85f5d1d3f004cf5817bb594fd8e6469 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 1. Current term is 2. Ops: []
I20260925 03:04:17.738150 25589 raft_consensus.cc:686] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 LEADER]: Becoming Leader. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:04:17.779083 25199 raft_consensus.cc:1261] T e6b6ec010caf4689b8c90449764f5cf9 P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260925 03:04:17.721659 25485 consensus_queue.cc:237] T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:17.792620 24436 leader_election.cc:302] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: 1bc1f2e62ab44af995165bd5116a9d43, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:17.796028 24436 leader_election.cc:302] T b4ebd2b043024a548005c8b6b00b0449 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: 1bc1f2e62ab44af995165bd5116a9d43, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:17.793200 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6b6ec010caf4689b8c90449764f5cf9" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:17.796317 25564 raft_consensus.cc:2726] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260925 03:04:17.760890 25589 consensus_queue.cc:237] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:17.804802 25587 raft_consensus.cc:2726] T b4ebd2b043024a548005c8b6b00b0449 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260925 03:04:17.810774 24465 raft_consensus.cc:3037] T 293ef69caf514ac98510a68495f54eda P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:17.847157 24548 raft_consensus.cc:1261] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Refusing update from remote peer 1bc1f2e62ab44af995165bd5116a9d43: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260925 03:04:17.807621 25550 consensus_queue.cc:1046] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.049s
W20260925 03:04:17.887197 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 74946.7us
W20260925 03:04:17.896440 25591 log.cc:924] Time spent T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.082s user 0.000s sys 0.001s
I20260925 03:04:17.886778 25547 consensus_queue.cc:1035] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.012s
I20260925 03:04:17.915333 24466 raft_consensus.cc:1261] T b4ebd2b043024a548005c8b6b00b0449 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Refusing update from remote peer 1bc1f2e62ab44af995165bd5116a9d43: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260925 03:04:17.901401 25165 leader_election.cc:302] T e6b6ec010caf4689b8c90449764f5cf9 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:17.914839 25584 consensus_queue.cc:1035] T e6b6ec010caf4689b8c90449764f5cf9 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.128s
I20260925 03:04:17.945600 25517 raft_consensus.cc:2726] T e6b6ec010caf4689b8c90449764f5cf9 P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260925 03:04:17.946069 24622 raft_consensus.cc:1226] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 1. Current term is 2. Ops: []
I20260925 03:04:17.961697 25564 raft_consensus.cc:3032] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 LEADER]: Stepping down as leader of term 1
I20260925 03:04:17.962152 25564 raft_consensus.cc:729] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:04:17.962894 25564 consensus_queue.cc:260] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
W20260925 03:04:17.956970 25165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 107857us
I20260925 03:04:17.961730 24613 raft_consensus.cc:2454] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 2.
I20260925 03:04:17.979750 24472 raft_consensus.cc:1261] T 7b8e0d260a884ecdb20f79de46c36b19 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260925 03:04:17.997929 24367 catalog_manager.cc:5495] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c reported cstate change: term changed from 1 to 2, leader changed from a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2) to eeab93abfdb141a895002ec9f312296c (127.22.193.3). New cstate: current_term: 2 leader_uuid: "eeab93abfdb141a895002ec9f312296c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: HEALTHY } } }
I20260925 03:04:17.998767 25196 raft_consensus.cc:2932] T e92f377a79354a6ebb4eec67303c13e3 P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } attrs { replace: true } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
W20260925 03:04:18.009851 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 73734.4us
I20260925 03:04:18.013826 24543 raft_consensus.cc:1261] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260925 03:04:17.982056 25563 consensus_queue.cc:1035] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.021s
I20260925 03:04:18.042961 25601 consensus_queue.cc:1035] T e6b6ec010caf4689b8c90449764f5cf9 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20260925 03:04:18.062613 24436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 120859us
I20260925 03:04:18.089449 25513 consensus_queue.cc:1046] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260925 03:04:18.090991 25584 raft_consensus.cc:3032] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 LEADER]: Stepping down as leader of term 1
I20260925 03:04:18.091416 25584 raft_consensus.cc:729] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:04:18.092499 25584 consensus_queue.cc:260] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:18.092027 24466 raft_consensus.cc:2932] T e92f377a79354a6ebb4eec67303c13e3 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } attrs { replace: true } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:18.094107 24436 leader_election.cc:302] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters: 1bc1f2e62ab44af995165bd5116a9d43
I20260925 03:04:18.095093 25604 raft_consensus.cc:2781] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 03:04:18.095577 25604 raft_consensus.cc:686] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 LEADER]: Becoming Leader. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:04:18.096465 25604 consensus_queue.cc:237] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:18.146924 25584 raft_consensus.cc:3037] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:18.173285 25517 consensus_queue.cc:1035] T b4ebd2b043024a548005c8b6b00b0449 P 1bc1f2e62ab44af995165bd5116a9d43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.006s
I20260925 03:04:18.178630 24371 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39374 (request call id 196) took 1017 ms. Trace:
I20260925 03:04:18.181362 25564 raft_consensus.cc:3037] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:18.181479 24371 rpcz_store.cc:276] 0925 03:04:17.160817 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:17.161052 (+ 235us) service_pool.cc:227] Handling call
0925 03:04:18.178566 (+1017514us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0925 03:04:17.363879 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0925 03:04:17.393649 (+ 29770us) write_op.cc:432] Acquiring schema lock in shared mode
0925 03:04:17.393724 (+ 75us) write_op.cc:435] Acquired schema lock
0925 03:04:17.393740 (+ 16us) tablet.cc:660] Decoding operations
0925 03:04:17.394470 (+ 730us) write_op.cc:620] Acquiring the partition lock for write op
0925 03:04:17.394594 (+ 124us) write_op.cc:641] Partition lock acquired for write op
0925 03:04:17.394652 (+ 58us) tablet.cc:683] Acquiring locks for 1 operations
0925 03:04:17.394904 (+ 252us) tablet.cc:699] Row locks acquired
0925 03:04:17.394919 (+ 15us) write_op.cc:260] PREPARE: finished
0925 03:04:17.395077 (+ 158us) write_op.cc:270] Start()
0925 03:04:17.395191 (+ 114us) write_op.cc:276] Timestamp: P: 1790305457395030 usec, L: 0
0925 03:04:17.395205 (+ 14us) op_driver.cc:331] REPLICATION: starting
0925 03:04:17.395910 (+ 705us) log.cc:842] Serialized 532 byte log entry
0925 03:04:17.886511 (+490601us) op_driver.cc:444] REPLICATION: finished
0925 03:04:18.036250 (+149739us) write_op.cc:301] APPLY: starting
0925 03:04:18.036340 (+ 90us) tablet.cc:1365] starting BulkCheckPresence
0925 03:04:18.036445 (+ 105us) tablet.cc:1368] finished BulkCheckPresence
0925 03:04:18.036589 (+ 144us) tablet.cc:1370] starting ApplyRowOperation cycle
0925 03:04:18.036874 (+ 285us) tablet.cc:1381] finished ApplyRowOperation cycle
0925 03:04:18.037032 (+ 158us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0925 03:04:18.037083 (+ 51us) write_op.cc:312] APPLY: finished
0925 03:04:18.056243 (+ 19160us) log.cc:842] Serialized 22 byte log entry
0925 03:04:18.063230 (+ 6987us) write_op.cc:489] Releasing partition, row and schema locks
0925 03:04:18.079699 (+ 16469us) write_op.cc:454] Released schema lock
0925 03:04:18.084609 (+ 4910us) write_op.cc:341] FINISH: Updating metrics
0925 03:04:18.121088 (+ 36479us) write_op.cc:489] Releasing partition, row and schema locks
0925 03:04:18.121117 (+ 29us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":43395,"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":130102,"apply.run_cpu_time_us":2563,"apply.run_wall_time_us":85153,"num_ops":1,"prepare.queue_time_us":157929,"prepare.run_cpu_time_us":18491,"prepare.run_wall_time_us":135976,"raft.queue_time_us":614746,"raft.run_cpu_time_us":34660,"raft.run_wall_time_us":212335,"replication_time_us":456425,"thread_start_us":393382,"threads_started":4,"wal-append.queue_time_us":144508}]]}
I20260925 03:04:18.189231 25533 raft_consensus.cc:2932] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } attrs { replace: true } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
W20260925 03:04:18.170529 25165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 85011.1us
I20260925 03:04:18.211776 24547 raft_consensus.cc:1261] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260925 03:04:18.214542 25505 consensus_queue.cc:1035] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260925 03:04:18.216590 24540 raft_consensus.cc:2932] T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } attrs { replace: true } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:18.233826 24367 catalog_manager.cc:5495] T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5 reported cstate change: term changed from 1 to 2, leader changed from eeab93abfdb141a895002ec9f312296c (127.22.193.3) to a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2). New cstate: current_term: 2 leader_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: UNKNOWN } } }
I20260925 03:04:18.236755 24367 catalog_manager.cc:5495] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 reported cstate change: term changed from 1 to 2, leader changed from a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2) to 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4). New cstate: current_term: 2 leader_uuid: "1bc1f2e62ab44af995165bd5116a9d43" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:18.265707 24622 raft_consensus.cc:1261] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260925 03:04:18.290002 25560 consensus_queue.cc:1035] T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
I20260925 03:04:18.334602 24464 raft_consensus.cc:3037] T 9e8af2641e124f9589c5ccfdac785f63 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:18.346462 24464 raft_consensus.cc:1261] T 9e8af2641e124f9589c5ccfdac785f63 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260925 03:04:18.349044 25560 consensus_queue.cc:1035] T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260925 03:04:18.345837 24369 catalog_manager.cc:5495] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5 reported cstate change: term changed from 1 to 2, leader changed from eeab93abfdb141a895002ec9f312296c (127.22.193.3) to 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1). New cstate: current_term: 2 leader_uuid: "869b0a8575b54d08ae2817fad5dd42d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: UNKNOWN } } }
I20260925 03:04:18.350975 24369 catalog_manager.cc:5495] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 reported cstate change: term changed from 1 to 2, leader changed from eeab93abfdb141a895002ec9f312296c (127.22.193.3) to 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1). New cstate: current_term: 2 leader_uuid: "869b0a8575b54d08ae2817fad5dd42d5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260925 03:04:18.358788 25529 raft_consensus.cc:482] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:18.358446 24367 catalog_manager.cc:5495] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c reported cstate change: config changed from index 2 to 3, 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "eeab93abfdb141a895002ec9f312296c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260925 03:04:18.359330 25529 raft_consensus.cc:504] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:18.365811 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abd5b8dc1b7e49e3aa9af900d73ff0e7" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:18.371886 25529 leader_election.cc:288] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:18.375322 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abd5b8dc1b7e49e3aa9af900d73ff0e7" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:18.375880 24472 raft_consensus.cc:2454] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 2.
I20260925 03:04:18.377126 24514 leader_election.cc:302] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:04:18.380299 25529 raft_consensus.cc:2781] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260925 03:04:18.380723 25529 raft_consensus.cc:482] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:18.381086 25529 raft_consensus.cc:3037] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:18.392848 24622 raft_consensus.cc:1261] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260925 03:04:18.393457 24618 consensus_queue.cc:237] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:18.394323 25584 consensus_queue.cc:1035] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260925 03:04:18.399219 25560 raft_consensus.cc:482] T 11bb46bce3564841b7674e6a693bd2e6 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:18.400241 25560 raft_consensus.cc:504] T 11bb46bce3564841b7674e6a693bd2e6 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:18.402870 25560 leader_election.cc:288] T 11bb46bce3564841b7674e6a693bd2e6 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:18.406229 24547 raft_consensus.cc:1261] T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260925 03:04:18.406005 24616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bb46bce3564841b7674e6a693bd2e6" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:18.406898 25196 raft_consensus.cc:1261] T e92f377a79354a6ebb4eec67303c13e3 P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260925 03:04:18.407485 24616 raft_consensus.cc:2379] T 11bb46bce3564841b7674e6a693bd2e6 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 2: Already voted for candidate eeab93abfdb141a895002ec9f312296c in this term.
I20260925 03:04:18.407910 25563 consensus_queue.cc:1035] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260925 03:04:18.408948 25572 consensus_queue.cc:1035] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260925 03:04:18.420583 25505 raft_consensus.cc:2932] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:18.422189 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bb46bce3564841b7674e6a693bd2e6" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:18.424062 24466 raft_consensus.cc:2454] T 11bb46bce3564841b7674e6a693bd2e6 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 2.
I20260925 03:04:18.425563 24514 leader_election.cc:302] T 11bb46bce3564841b7674e6a693bd2e6 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:04:18.426698 25560 raft_consensus.cc:2781] T 11bb46bce3564841b7674e6a693bd2e6 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 03:04:18.427063 25560 raft_consensus.cc:482] T 11bb46bce3564841b7674e6a693bd2e6 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:18.427402 25560 raft_consensus.cc:3037] T 11bb46bce3564841b7674e6a693bd2e6 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:18.429275 25604 raft_consensus.cc:482] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:18.429795 25604 raft_consensus.cc:504] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:18.429710 25529 raft_consensus.cc:504] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:18.433979 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:18.435139 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abd5b8dc1b7e49e3aa9af900d73ff0e7" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:18.435753 24466 raft_consensus.cc:3037] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:18.436061 25604 leader_election.cc:288] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:18.441968 25196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:18.442734 25196 raft_consensus.cc:2454] T 7e49248643df4131a1c42e74244e05c2 P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 LEARNER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 1.
I20260925 03:04:18.443006 24618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:18.442203 25199 raft_consensus.cc:2932] T e92f377a79354a6ebb4eec67303c13e3 P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:18.443634 24618 raft_consensus.cc:2454] T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 2.
I20260925 03:04:18.445395 25529 leader_election.cc:288] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:18.446513 24466 raft_consensus.cc:2454] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 3.
I20260925 03:04:18.448048 24514 leader_election.cc:302] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:04:18.449065 24436 leader_election.cc:302] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:18.449837 25516 raft_consensus.cc:2781] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 03:04:18.450349 25516 raft_consensus.cc:686] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 LEADER]: Becoming Leader. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:04:18.451119 25516 consensus_queue.cc:237] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:18.451534 24547 raft_consensus.cc:2932] T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:18.454154 25604 raft_consensus.cc:2781] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260925 03:04:18.454553 25604 raft_consensus.cc:482] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:18.454886 25604 raft_consensus.cc:3037] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:18.458069 24618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abd5b8dc1b7e49e3aa9af900d73ff0e7" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:18.468075 24545 raft_consensus.cc:1261] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260925 03:04:18.468425 24355 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet e92f377a79354a6ebb4eec67303c13e3 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260925 03:04:18.469995 25485 raft_consensus.cc:482] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:18.470498 25485 raft_consensus.cc:504] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:18.472401 25485 leader_election.cc:288] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:18.474047 25560 raft_consensus.cc:504] T 11bb46bce3564841b7674e6a693bd2e6 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:18.477334 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bb46bce3564841b7674e6a693bd2e6" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:18.477092 25584 consensus_queue.cc:1035] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260925 03:04:18.485132 24616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bb46bce3564841b7674e6a693bd2e6" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:18.485977 24616 raft_consensus.cc:2379] T 11bb46bce3564841b7674e6a693bd2e6 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 2: Already voted for candidate eeab93abfdb141a895002ec9f312296c in this term.
I20260925 03:04:18.488204 24367 catalog_manager.cc:5495] T e92f377a79354a6ebb4eec67303c13e3 P 1bc1f2e62ab44af995165bd5116a9d43 reported cstate change: config changed from index 3 to 4, VOTER 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1) evicted. New cstate: current_term: 1 leader_uuid: "eeab93abfdb141a895002ec9f312296c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:18.494429 24472 raft_consensus.cc:2454] T 11bb46bce3564841b7674e6a693bd2e6 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 2.
I20260925 03:04:18.494676 25485 raft_consensus.cc:482] T c88cbd39e01f44628a094af516be2833 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:18.495146 25485 raft_consensus.cc:504] T c88cbd39e01f44628a094af516be2833 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:18.497247 25485 leader_election.cc:288] T c88cbd39e01f44628a094af516be2833 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:18.484740 25560 leader_election.cc:288] T 11bb46bce3564841b7674e6a693bd2e6 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:18.498589 24514 leader_election.cc:302] T 11bb46bce3564841b7674e6a693bd2e6 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:04:18.499330 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c88cbd39e01f44628a094af516be2833" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:18.504380 25485 raft_consensus.cc:2781] T 11bb46bce3564841b7674e6a693bd2e6 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 03:04:18.504925 25485 raft_consensus.cc:686] T 11bb46bce3564841b7674e6a693bd2e6 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 LEADER]: Becoming Leader. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:04:18.507534 25485 consensus_queue.cc:237] T 11bb46bce3564841b7674e6a693bd2e6 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:18.513643 25199 raft_consensus.cc:1261] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260925 03:04:18.513923 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9c33fd100ff44c1a3247319a324a044" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:18.515882 25539 consensus_queue.cc:1035] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260925 03:04:18.527112 24456 tablet_service.cc:1514] Processing DeleteTablet for tablet e92f377a79354a6ebb4eec67303c13e3 with delete_type TABLET_DATA_TOMBSTONED (TS 869b0a8575b54d08ae2817fad5dd42d5 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:55552
I20260925 03:04:18.521927 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c88cbd39e01f44628a094af516be2833" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:18.529039 24472 raft_consensus.cc:2454] T c88cbd39e01f44628a094af516be2833 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 2.
I20260925 03:04:18.532857 24514 leader_election.cc:302] T c88cbd39e01f44628a094af516be2833 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:04:18.535439 25529 raft_consensus.cc:2781] T c88cbd39e01f44628a094af516be2833 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260925 03:04:18.567087 25529 raft_consensus.cc:482] T c88cbd39e01f44628a094af516be2833 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:18.567735 25529 raft_consensus.cc:3037] T c88cbd39e01f44628a094af516be2833 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:18.515848 25604 raft_consensus.cc:504] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:18.546676 25561 raft_consensus.cc:482] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:18.555037 25516 raft_consensus.cc:482] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:18.555292 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9c33fd100ff44c1a3247319a324a044" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:18.587762 24466 raft_consensus.cc:2454] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 2.
I20260925 03:04:18.586491 25516 raft_consensus.cc:504] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:18.583376 25561 raft_consensus.cc:504] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } }
I20260925 03:04:18.606775 25634 tablet_replica.cc:330] T e92f377a79354a6ebb4eec67303c13e3 P 869b0a8575b54d08ae2817fad5dd42d5: stopping tablet replica
I20260925 03:04:18.607980 25634 raft_consensus.cc:2229] T e92f377a79354a6ebb4eec67303c13e3 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 03:04:18.608294 25561 leader_election.cc:288] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:18.609975 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367277daf8d04778b3fa4ed94d71580d" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:18.610630 24547 raft_consensus.cc:2454] T 367277daf8d04778b3fa4ed94d71580d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 1.
I20260925 03:04:18.611501 24437 leader_election.cc:302] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:04:18.602645 25555 raft_consensus.cc:482] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:18.612509 25555 raft_consensus.cc:504] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:18.613368 25558 raft_consensus.cc:2781] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260925 03:04:18.613754 25558 raft_consensus.cc:482] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:18.614055 25558 raft_consensus.cc:3037] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:18.618229 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:18.619051 24472 raft_consensus.cc:2354] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 2. Current term is 3.
I20260925 03:04:18.619379 24616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367277daf8d04778b3fa4ed94d71580d" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:18.620139 24616 raft_consensus.cc:2454] T 367277daf8d04778b3fa4ed94d71580d P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 2.
I20260925 03:04:18.620362 24514 leader_election.cc:398] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187) with higher term. Message: Invalid argument: T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 2. Current term is 3.
I20260925 03:04:18.620723 24514 leader_election.cc:401] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260925 03:04:18.621196 25558 raft_consensus.cc:504] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } }
I20260925 03:04:18.623250 25516 leader_election.cc:288] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:18.622145 25485 raft_consensus.cc:3037] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Advancing to term 3
I20260925 03:04:18.624042 24367 catalog_manager.cc:5495] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P a04a37fc4aa7460cbfdbeeb4e451e1f5 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: UNKNOWN } } }
I20260925 03:04:18.625473 24514 leader_election.cc:302] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:04:18.627935 24367 catalog_manager.cc:5495] T 11bb46bce3564841b7674e6a693bd2e6 P a04a37fc4aa7460cbfdbeeb4e451e1f5 reported cstate change: term changed from 1 to 2, leader changed from 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1) to a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2). New cstate: current_term: 2 leader_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: UNKNOWN } } }
I20260925 03:04:18.627705 25634 raft_consensus.cc:2258] T e92f377a79354a6ebb4eec67303c13e3 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 03:04:18.628249 25560 raft_consensus.cc:2781] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260925 03:04:18.634964 25560 raft_consensus.cc:482] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:18.635867 25560 raft_consensus.cc:3037] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:18.636808 25572 raft_consensus.cc:482] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:18.637308 25572 raft_consensus.cc:504] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:18.628923 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5332a5d329493991dff994e70ea92c" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:18.638833 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5332a5d329493991dff994e70ea92c" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:18.639508 24466 raft_consensus.cc:2454] T 4e5332a5d329493991dff994e70ea92c P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 2.
I20260925 03:04:18.640470 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5f9b503b0ab43999d582015a367441e" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:18.640929 25196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:18.641095 24545 raft_consensus.cc:2454] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 2.
I20260925 03:04:18.633780 25563 raft_consensus.cc:482] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:18.641845 25563 raft_consensus.cc:504] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:18.642282 24586 leader_election.cc:302] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:18.643062 25557 raft_consensus.cc:482] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:18.643708 24514 leader_election.cc:302] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:04:18.643893 24613 raft_consensus.cc:2454] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 1.
I20260925 03:04:18.644208 25535 raft_consensus.cc:482] T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:18.643749 25563 leader_election.cc:288] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:18.644739 25535 raft_consensus.cc:504] T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:18.645120 25563 raft_consensus.cc:2781] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 03:04:18.645386 25516 raft_consensus.cc:2781] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260925 03:04:18.645649 25563 raft_consensus.cc:482] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting leader election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:18.646469 25516 raft_consensus.cc:482] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:18.646778 25535 leader_election.cc:288] T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:18.643513 25557 raft_consensus.cc:504] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:18.646898 25516 raft_consensus.cc:3037] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:18.646194 25563 raft_consensus.cc:3037] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:18.650178 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06f4a7eaf6534e4986c9fd35911038e4" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:18.650915 24464 raft_consensus.cc:2379] T 06f4a7eaf6534e4986c9fd35911038e4 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c in current term 2: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:04:18.651888 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06f4a7eaf6534e4986c9fd35911038e4" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:18.651950 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7691c6b475428c82d6181d693d26f3" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:18.652482 24545 raft_consensus.cc:2454] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 2.
I20260925 03:04:18.652649 24547 raft_consensus.cc:2454] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 2.
I20260925 03:04:18.632287 24618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:18.653573 25557 leader_election.cc:288] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:18.654148 25533 raft_consensus.cc:482] T c88cbd39e01f44628a094af516be2833 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:18.654593 25540 raft_consensus.cc:482] T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:18.654491 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7691c6b475428c82d6181d693d26f3" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:18.655407 24469 raft_consensus.cc:2379] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c in current term 2: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:04:18.655112 25540 raft_consensus.cc:504] T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:18.641541 25196 raft_consensus.cc:2454] T 7e49248643df4131a1c42e74244e05c2 P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 LEARNER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 1.
I20260925 03:04:18.656291 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5332a5d329493991dff994e70ea92c" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:18.657335 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5f9b503b0ab43999d582015a367441e" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:18.639213 25572 leader_election.cc:288] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:18.658118 24466 raft_consensus.cc:2379] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c in current term 2: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:04:18.641732 25505 raft_consensus.cc:482] T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:18.658679 25505 raft_consensus.cc:504] T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:18.654181 24618 raft_consensus.cc:2379] T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 2: Already voted for candidate eeab93abfdb141a895002ec9f312296c in this term.
I20260925 03:04:18.659224 25540 leader_election.cc:288] T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:18.658221 25572 raft_consensus.cc:482] T 0b5df303cd6a4d59aca5c18bbbff24ed P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:18.654655 25533 raft_consensus.cc:504] T c88cbd39e01f44628a094af516be2833 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:18.660810 25540 raft_consensus.cc:482] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:18.661942 25540 raft_consensus.cc:504] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:18.662767 25533 leader_election.cc:288] T c88cbd39e01f44628a094af516be2833 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:18.663262 25533 raft_consensus.cc:482] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:18.663658 25533 raft_consensus.cc:504] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:18.666607 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9c33fd100ff44c1a3247319a324a044" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:18.660728 25505 leader_election.cc:288] T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:18.633818 25555 leader_election.cc:288] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:18.667991 25557 raft_consensus.cc:482] T 74443a0a1dc34d88a2f42e3b722db405 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:18.661429 25572 raft_consensus.cc:504] T 0b5df303cd6a4d59aca5c18bbbff24ed P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:18.668500 25557 raft_consensus.cc:504] T 74443a0a1dc34d88a2f42e3b722db405 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:18.670156 25572 leader_election.cc:288] T 0b5df303cd6a4d59aca5c18bbbff24ed P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:18.670388 25557 leader_election.cc:288] T 74443a0a1dc34d88a2f42e3b722db405 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:18.670924 25557 raft_consensus.cc:482] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:18.671350 25557 raft_consensus.cc:504] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:18.672587 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5332a5d329493991dff994e70ea92c" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:18.673133 25540 leader_election.cc:288] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:18.674881 25634 ts_tablet_manager.cc:1905] T e92f377a79354a6ebb4eec67303c13e3 P 869b0a8575b54d08ae2817fad5dd42d5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260925 03:04:18.677718 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74443a0a1dc34d88a2f42e3b722db405" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:18.673368 24472 raft_consensus.cc:2379] T 4e5332a5d329493991dff994e70ea92c P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c in current term 2: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:04:18.678462 24613 raft_consensus.cc:2454] T 74443a0a1dc34d88a2f42e3b722db405 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 1.
I20260925 03:04:18.680271 25535 raft_consensus.cc:482] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:18.667500 24586 leader_election.cc:302] T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:18.682765 24586 leader_election.cc:302] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:04:18.686831 25533 leader_election.cc:288] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:18.687036 25516 raft_consensus.cc:504] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:18.689073 24540 raft_consensus.cc:2354] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c for earlier term 2. Current term is 3.
I20260925 03:04:18.689707 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9c33fd100ff44c1a3247319a324a044" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:18.690608 24464 raft_consensus.cc:2379] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c in current term 2: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:04:18.690780 24465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b5df303cd6a4d59aca5c18bbbff24ed" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:18.690536 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c88cbd39e01f44628a094af516be2833" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:18.691547 24465 raft_consensus.cc:2379] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c in current term 2: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:04:18.691888 24469 raft_consensus.cc:2379] T c88cbd39e01f44628a094af516be2833 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c in current term 2: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:04:18.691617 25516 leader_election.cc:288] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:18.688639 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74443a0a1dc34d88a2f42e3b722db405" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:18.692858 24465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74443a0a1dc34d88a2f42e3b722db405" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:18.694336 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b5df303cd6a4d59aca5c18bbbff24ed" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:18.693545 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c88cbd39e01f44628a094af516be2833" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:18.693221 24472 raft_consensus.cc:2454] T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 2.
I20260925 03:04:18.697009 24586 leader_election.cc:398] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Vote denied by peer a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389) with higher term. Message: Invalid argument: T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c for earlier term 2. Current term is 3.
I20260925 03:04:18.697324 24586 leader_election.cc:401] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260925 03:04:18.698210 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e725b82985746b39c0cc8f671c95353" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:18.695294 24546 raft_consensus.cc:2454] T 0b5df303cd6a4d59aca5c18bbbff24ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 2.
I20260925 03:04:18.702770 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f1fac028a144058b92331c0d9b44ebf" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:18.698884 24543 raft_consensus.cc:2454] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 2.
I20260925 03:04:18.703577 24472 raft_consensus.cc:2454] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 2.
I20260925 03:04:18.704769 24514 leader_election.cc:302] T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:04:18.709153 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e725b82985746b39c0cc8f671c95353" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:18.709473 24514 leader_election.cc:302] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:04:18.710873 24586 leader_election.cc:302] T 0b5df303cd6a4d59aca5c18bbbff24ed P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:04:18.712179 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74443a0a1dc34d88a2f42e3b722db405" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:18.714617 24472 raft_consensus.cc:2379] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c in current term 2: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:04:18.712725 24548 raft_consensus.cc:2454] T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 2.
I20260925 03:04:18.681180 24616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f1fac028a144058b92331c0d9b44ebf" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:18.715296 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5332a5d329493991dff994e70ea92c" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:18.718576 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367277daf8d04778b3fa4ed94d71580d" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:18.680908 25535 raft_consensus.cc:504] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:18.720748 24548 raft_consensus.cc:3037] T 367277daf8d04778b3fa4ed94d71580d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Advancing to term 3
I20260925 03:04:18.710481 25533 raft_consensus.cc:2781] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 03:04:18.702111 25505 raft_consensus.cc:2781] T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 03:04:18.726441 24616 raft_consensus.cc:2379] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 2: Already voted for candidate eeab93abfdb141a895002ec9f312296c in this term.
I20260925 03:04:18.726311 24469 raft_consensus.cc:3037] T 4e5332a5d329493991dff994e70ea92c P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:18.710348 25572 raft_consensus.cc:3037] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Advancing to term 3
I20260925 03:04:18.715583 25555 raft_consensus.cc:2781] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 03:04:18.720077 25540 raft_consensus.cc:2781] T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260925 03:04:18.720737 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:18.751565 24548 raft_consensus.cc:2454] T 367277daf8d04778b3fa4ed94d71580d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 3.
I20260925 03:04:18.724385 25547 raft_consensus.cc:482] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:18.732877 24465 raft_consensus.cc:2379] T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c in current term 2: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:04:18.753576 24437 leader_election.cc:302] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:04:18.756158 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abd5b8dc1b7e49e3aa9af900d73ff0e7" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:18.775523 24464 raft_consensus.cc:2354] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c for earlier term 2. Current term is 3.
I20260925 03:04:18.768591 25635 raft_consensus.cc:482] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:18.778358 25635 raft_consensus.cc:504] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:18.778493 25563 raft_consensus.cc:504] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:18.780053 24618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5332a5d329493991dff994e70ea92c" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:18.780417 25635 leader_election.cc:288] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:18.774626 25560 raft_consensus.cc:504] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:18.782140 24545 raft_consensus.cc:2354] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c for earlier term 2. Current term is 3.
I20260925 03:04:18.783878 25563 leader_election.cc:288] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:18.784482 25563 raft_consensus.cc:482] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:18.752424 25535 leader_election.cc:288] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:18.767527 24613 raft_consensus.cc:1261] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260925 03:04:18.767117 25547 raft_consensus.cc:504] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:18.767570 25604 leader_election.cc:288] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:18.748251 25555 raft_consensus.cc:482] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:18.768469 25636 raft_consensus.cc:482] T 442d4a29b1d743599f4b6b4871e26668 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:18.727280 25533 raft_consensus.cc:482] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting leader election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:18.771569 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:18.772882 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367277daf8d04778b3fa4ed94d71580d" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:18.774341 25199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43"
I20260925 03:04:18.829614 24622 raft_consensus.cc:3037] T 367277daf8d04778b3fa4ed94d71580d P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:18.784938 24465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5f9b503b0ab43999d582015a367441e" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:18.832362 24465 raft_consensus.cc:2379] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c in current term 2: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:04:18.787012 25485 raft_consensus.cc:2726] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187) with higher term. Message: Invalid argument: T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 2. Current term is 3.
I20260925 03:04:18.788280 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ac285175464fbda67211d9574db415" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:18.788532 25584 raft_consensus.cc:482] T c85f5d1d3f004cf5817bb594fd8e6469 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:18.789034 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9c33fd100ff44c1a3247319a324a044" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:18.833695 24464 raft_consensus.cc:2454] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 2.
I20260925 03:04:18.833943 24466 raft_consensus.cc:3037] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:18.836153 25634 ts_tablet_manager.cc:1918] T e92f377a79354a6ebb4eec67303c13e3 P 869b0a8575b54d08ae2817fad5dd42d5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260925 03:04:18.836642 25634 log.cc:1196] T e92f377a79354a6ebb4eec67303c13e3 P 869b0a8575b54d08ae2817fad5dd42d5: Deleting WAL directory at /tmp/dist-test-tasklAj8hX/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790305368086980-23300-0/minicluster-data/ts-0-root/wals/e92f377a79354a6ebb4eec67303c13e3
I20260925 03:04:18.837291 25602 raft_consensus.cc:482] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:18.787868 24616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ac285175464fbda67211d9574db415" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:18.789968 25560 leader_election.cc:288] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:18.790866 24588 leader_election.cc:398] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Vote denied by peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187) with higher term. Message: Invalid argument: T abd5b8dc1b7e49e3aa9af900d73ff0e7 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c for earlier term 2. Current term is 3.
I20260925 03:04:18.755890 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35992316cb8d4c028333388eff02ecde" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:18.837774 25602 raft_consensus.cc:504] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:18.799721 25540 raft_consensus.cc:482] T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260925 03:04:18.823611 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 112835us
I20260925 03:04:18.732667 25505 raft_consensus.cc:482] T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting leader election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:18.759164 25558 leader_election.cc:288] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:18.842366 25540 raft_consensus.cc:3037] T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:18.842674 25505 raft_consensus.cc:3037] T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:18.785845 25490 raft_consensus.cc:2781] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 03:04:18.809010 25557 leader_election.cc:288] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:18.830821 24621 raft_consensus.cc:3037] T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:18.843894 25490 raft_consensus.cc:686] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 LEADER]: Becoming Leader. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:04:18.844811 25490 consensus_queue.cc:237] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } }
I20260925 03:04:18.793401 24614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9c33fd100ff44c1a3247319a324a044" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:18.838295 24588 leader_election.cc:401] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260925 03:04:18.838685 24472 raft_consensus.cc:2379] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c in current term 2: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:04:18.833724 25584 raft_consensus.cc:504] T c85f5d1d3f004cf5817bb594fd8e6469 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:18.827244 25555 raft_consensus.cc:3037] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:18.849273 25584 leader_election.cc:288] T c85f5d1d3f004cf5817bb594fd8e6469 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:18.839627 25547 leader_election.cc:288] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:18.829437 25533 raft_consensus.cc:3037] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:18.840320 24469 raft_consensus.cc:2454] T 4e5332a5d329493991dff994e70ea92c P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 3.
I20260925 03:04:18.857280 24622 raft_consensus.cc:2454] T 367277daf8d04778b3fa4ed94d71580d P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 3.
I20260925 03:04:18.844750 24466 raft_consensus.cc:2454] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 3.
W20260925 03:04:18.859508 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 68794.9us
I20260925 03:04:18.838337 24616 raft_consensus.cc:2379] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 2: Already voted for candidate eeab93abfdb141a895002ec9f312296c in this term.
I20260925 03:04:18.840829 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5aec38e66a947c681e6c2a72b05e6ca" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:18.829702 25199 raft_consensus.cc:3037] T 7e49248643df4131a1c42e74244e05c2 P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 LEARNER]: Advancing to term 3
I20260925 03:04:18.861912 25557 raft_consensus.cc:3037] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Advancing to term 3
I20260925 03:04:18.847594 24614 raft_consensus.cc:3037] T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Advancing to term 3
I20260925 03:04:18.863758 25533 raft_consensus.cc:504] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:18.865542 25485 raft_consensus.cc:482] T 442d4a29b1d743599f4b6b4871e26668 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:18.860646 24620 raft_consensus.cc:2379] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1bc1f2e62ab44af995165bd5116a9d43 in current term 2: Already voted for candidate eeab93abfdb141a895002ec9f312296c in this term.
I20260925 03:04:18.820828 25635 raft_consensus.cc:482] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:18.866036 25485 raft_consensus.cc:504] T 442d4a29b1d743599f4b6b4871e26668 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:18.866334 25635 raft_consensus.cc:504] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:18.867805 25533 leader_election.cc:288] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:18.867992 25485 leader_election.cc:288] T 442d4a29b1d743599f4b6b4871e26668 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:18.868366 25635 leader_election.cc:288] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:18.829763 25636 raft_consensus.cc:504] T 442d4a29b1d743599f4b6b4871e26668 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:18.870607 24616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442d4a29b1d743599f4b6b4871e26668" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:18.871281 24616 raft_consensus.cc:2454] T 442d4a29b1d743599f4b6b4871e26668 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 1.
I20260925 03:04:18.829921 25604 raft_consensus.cc:482] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:18.872475 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7691c6b475428c82d6181d693d26f3" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:18.872786 25539 raft_consensus.cc:482] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:18.854027 25560 raft_consensus.cc:482] T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:18.825155 25535 raft_consensus.cc:2781] T 0b5df303cd6a4d59aca5c18bbbff24ed P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 03:04:18.873468 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b627ecb4a5384ef392aa695e06599c39" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:18.873711 25535 raft_consensus.cc:482] T 0b5df303cd6a4d59aca5c18bbbff24ed P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting leader election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:18.874084 24622 raft_consensus.cc:2454] T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 2.
I20260925 03:04:18.873304 25539 raft_consensus.cc:504] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:18.874162 25535 raft_consensus.cc:3037] T 0b5df303cd6a4d59aca5c18bbbff24ed P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:18.875864 25539 leader_election.cc:288] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:18.873333 24466 raft_consensus.cc:2379] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c in current term 2: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:04:18.872278 25604 raft_consensus.cc:504] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:18.879150 25199 raft_consensus.cc:2454] T 7e49248643df4131a1c42e74244e05c2 P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 LEARNER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 3.
I20260925 03:04:18.880326 24511 leader_election.cc:302] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:18.883797 24614 raft_consensus.cc:2454] T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 3.
I20260925 03:04:18.886029 24586 leader_election.cc:302] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:04:18.886978 25637 raft_consensus.cc:2781] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 03:04:18.864506 25540 raft_consensus.cc:504] T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:18.888113 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442d4a29b1d743599f4b6b4871e26668" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:18.889232 24466 raft_consensus.cc:2379] T 442d4a29b1d743599f4b6b4871e26668 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c in current term 2: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:04:18.891161 25516 raft_consensus.cc:2781] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260925 03:04:18.891603 25516 raft_consensus.cc:482] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:18.891935 25516 raft_consensus.cc:3037] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:18.893285 24358 catalog_manager.cc:4841] TS 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187): tablet e92f377a79354a6ebb4eec67303c13e3 (table test-workload [id=f20271173d58474897a63e58ef5a856d]) successfully deleted
I20260925 03:04:18.871572 25636 leader_election.cc:288] T 442d4a29b1d743599f4b6b4871e26668 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:18.899718 24614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c85f5d1d3f004cf5817bb594fd8e6469" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:18.890950 25540 leader_election.cc:288] T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:18.900908 25540 raft_consensus.cc:482] T 37a157996e944db2906110d87ecb0c54 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:18.880581 25640 raft_consensus.cc:482] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:18.901424 25540 raft_consensus.cc:504] T 37a157996e944db2906110d87ecb0c54 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:18.903213 25539 consensus_queue.cc:1035] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260925 03:04:18.879786 25635 raft_consensus.cc:482] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:18.905982 25540 leader_election.cc:288] T 37a157996e944db2906110d87ecb0c54 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:18.905041 25635 raft_consensus.cc:504] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:18.906596 25540 raft_consensus.cc:482] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:18.879671 25560 raft_consensus.cc:504] T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:18.909160 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5aec38e66a947c681e6c2a72b05e6ca" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:18.907852 25540 raft_consensus.cc:504] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:18.909902 24466 raft_consensus.cc:2454] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 2.
I20260925 03:04:18.910811 24621 raft_consensus.cc:2454] T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 3.
I20260925 03:04:18.900409 25529 raft_consensus.cc:504] T c88cbd39e01f44628a094af516be2833 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:18.914017 24547 raft_consensus.cc:2354] T c88cbd39e01f44628a094af516be2833 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c for earlier term 2. Current term is 3.
I20260925 03:04:18.914657 25557 raft_consensus.cc:2726] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187) with higher term. Message: Invalid argument: T abd5b8dc1b7e49e3aa9af900d73ff0e7 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c for earlier term 2. Current term is 3.
I20260925 03:04:18.914577 25540 leader_election.cc:288] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:18.910928 25560 leader_election.cc:288] T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:18.887384 24586 leader_election.cc:302] T 74443a0a1dc34d88a2f42e3b722db405 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:04:18.911691 24614 raft_consensus.cc:2454] T c85f5d1d3f004cf5817bb594fd8e6469 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 2.
I20260925 03:04:18.901865 25640 raft_consensus.cc:504] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:04:18.918041 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 88672.6us
I20260925 03:04:18.918377 24436 leader_election.cc:302] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:18.919656 24514 leader_election.cc:302] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:04:18.920420 24586 leader_election.cc:398] T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Vote denied by peer a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389) with higher term. Message: Invalid argument: T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c for earlier term 2. Current term is 3.
I20260925 03:04:18.920727 24586 leader_election.cc:401] T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260925 03:04:18.921264 24436 leader_election.cc:302] T c85f5d1d3f004cf5817bb594fd8e6469 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:18.921885 24586 leader_election.cc:398] T c88cbd39e01f44628a094af516be2833 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Vote denied by peer a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389) with higher term. Message: Invalid argument: T c88cbd39e01f44628a094af516be2833 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c for earlier term 2. Current term is 3.
I20260925 03:04:18.922170 24586 leader_election.cc:401] T c88cbd39e01f44628a094af516be2833 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260925 03:04:18.923918 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abd5b8dc1b7e49e3aa9af900d73ff0e7" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:18.924425 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5f9b503b0ab43999d582015a367441e" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:18.925091 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35992316cb8d4c028333388eff02ecde" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:18.925742 24548 raft_consensus.cc:2454] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 2.
I20260925 03:04:18.894093 25637 raft_consensus.cc:482] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting leader election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:18.930840 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "420163a6cdce412ebc21c25fe3835638" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:18.912537 25604 leader_election.cc:288] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:18.931969 25637 raft_consensus.cc:3037] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:18.824255 25563 raft_consensus.cc:504] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:18.929598 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442d4a29b1d743599f4b6b4871e26668" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:18.926687 25645 raft_consensus.cc:2781] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 03:04:18.926926 25642 raft_consensus.cc:2781] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 03:04:18.931404 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7691c6b475428c82d6181d693d26f3" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:18.937402 24545 raft_consensus.cc:2454] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 2.
I20260925 03:04:18.933990 24543 raft_consensus.cc:2454] T 442d4a29b1d743599f4b6b4871e26668 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 2.
I20260925 03:04:18.911093 25635 leader_election.cc:288] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:18.941220 24514 leader_election.cc:302] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:04:18.942544 24514 leader_election.cc:302] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:04:18.947472 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442d4a29b1d743599f4b6b4871e26668" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:18.948236 24472 raft_consensus.cc:2454] T 442d4a29b1d743599f4b6b4871e26668 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 2.
I20260925 03:04:18.952867 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b627ecb4a5384ef392aa695e06599c39" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:18.953562 24472 raft_consensus.cc:2454] T b627ecb4a5384ef392aa695e06599c39 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 2.
I20260925 03:04:18.953753 24586 leader_election.cc:302] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:04:18.921646 24616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b030d84c670b48d5bff337ccc0c77626" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:18.930830 25639 raft_consensus.cc:2781] T 74443a0a1dc34d88a2f42e3b722db405 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 03:04:18.938661 25645 raft_consensus.cc:482] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:18.938215 24540 raft_consensus.cc:2454] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 3.
W20260925 03:04:18.956424 24511 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 76175.4us
I20260925 03:04:18.987713 25560 raft_consensus.cc:482] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:18.958112 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74443a0a1dc34d88a2f42e3b722db405" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:18.959233 24464 raft_consensus.cc:1261] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260925 03:04:18.939277 25636 raft_consensus.cc:3037] T c88cbd39e01f44628a094af516be2833 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Advancing to term 3
I20260925 03:04:18.988751 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c85f5d1d3f004cf5817bb594fd8e6469" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:18.989897 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74443a0a1dc34d88a2f42e3b722db405" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:18.990216 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37a157996e944db2906110d87ecb0c54" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:18.991079 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35992316cb8d4c028333388eff02ecde" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:18.991540 24546 raft_consensus.cc:2454] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 2.
I20260925 03:04:18.991677 24619 raft_consensus.cc:2454] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 1.
I20260925 03:04:18.992219 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c23337a7cb74184a0e57b7095cb6a11" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:18.938330 25533 raft_consensus.cc:482] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:18.958979 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60d45b7d467d4e1ab92367f6d7d52b15" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:18.992980 24617 raft_consensus.cc:2379] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 2: Already voted for candidate eeab93abfdb141a895002ec9f312296c in this term.
I20260925 03:04:18.993106 24468 raft_consensus.cc:2454] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 2.
I20260925 03:04:18.992997 25533 raft_consensus.cc:504] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:18.993983 24617 raft_consensus.cc:1261] T 11bb46bce3564841b7674e6a693bd2e6 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260925 03:04:18.931854 25165 leader_election.cc:302] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:04:18.940208 24621 raft_consensus.cc:2454] T 420163a6cdce412ebc21c25fe3835638 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 2.
I20260925 03:04:18.933444 25561 raft_consensus.cc:2781] T c85f5d1d3f004cf5817bb594fd8e6469 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260925 03:04:18.995332 25561 raft_consensus.cc:482] T c85f5d1d3f004cf5817bb594fd8e6469 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:18.933573 25638 raft_consensus.cc:2726] T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389) with higher term. Message: Invalid argument: T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c for earlier term 2. Current term is 3.
I20260925 03:04:18.982295 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c23337a7cb74184a0e57b7095cb6a11" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
W20260925 03:04:18.996269 25165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 64602.8us
I20260925 03:04:18.995283 24472 raft_consensus.cc:3037] T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:18.988529 25560 raft_consensus.cc:504] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } }
I20260925 03:04:18.994894 24615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60d45b7d467d4e1ab92367f6d7d52b15" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:18.997725 25547 raft_consensus.cc:2781] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 03:04:18.998123 25547 raft_consensus.cc:482] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:18.998227 25638 raft_consensus.cc:482] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:18.998548 25547 raft_consensus.cc:3037] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:18.999470 25560 leader_election.cc:288] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:18.996430 24620 raft_consensus.cc:2454] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 2.
I20260925 03:04:18.999146 25638 raft_consensus.cc:504] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:18.994866 25533 leader_election.cc:288] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:19.000705 25533 raft_consensus.cc:2781] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 03:04:18.995963 25561 raft_consensus.cc:3037] T c85f5d1d3f004cf5817bb594fd8e6469 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:19.001066 25533 raft_consensus.cc:482] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting leader election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:19.001658 25533 raft_consensus.cc:3037] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:18.997815 24615 raft_consensus.cc:2454] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 1.
I20260925 03:04:19.001046 25199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:19.004774 25638 leader_election.cc:288] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:19.005299 25199 raft_consensus.cc:2454] T 29637f5a4a504ce6ba29033ccd0cd716 P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 LEARNER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 1.
I20260925 03:04:19.007680 25547 raft_consensus.cc:504] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:19.011270 25636 raft_consensus.cc:2726] T c88cbd39e01f44628a094af516be2833 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389) with higher term. Message: Invalid argument: T c88cbd39e01f44628a094af516be2833 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c for earlier term 2. Current term is 3.
I20260925 03:04:19.011214 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "420163a6cdce412ebc21c25fe3835638" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:19.012043 24545 raft_consensus.cc:2454] T 420163a6cdce412ebc21c25fe3835638 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 1.
I20260925 03:04:19.012086 25636 raft_consensus.cc:482] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:19.013239 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c23337a7cb74184a0e57b7095cb6a11" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:19.012831 25636 raft_consensus.cc:504] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:19.013814 24545 raft_consensus.cc:2454] T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 1.
I20260925 03:04:19.014863 24472 raft_consensus.cc:2454] T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 3.
I20260925 03:04:19.011672 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b030d84c670b48d5bff337ccc0c77626" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:19.015419 25560 raft_consensus.cc:482] T b030d84c670b48d5bff337ccc0c77626 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:19.014317 24511 leader_election.cc:302] T 442d4a29b1d743599f4b6b4871e26668 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:19.015987 25560 raft_consensus.cc:504] T b030d84c670b48d5bff337ccc0c77626 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:18.955113 25540 raft_consensus.cc:2781] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 03:04:19.020414 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60d45b7d467d4e1ab92367f6d7d52b15" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:19.020918 25540 raft_consensus.cc:686] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 LEADER]: Becoming Leader. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:04:19.021224 24471 raft_consensus.cc:2379] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c in current term 2: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:04:19.021508 25636 leader_election.cc:288] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:18.972374 25572 raft_consensus.cc:2726] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389) with higher term. Message: Invalid argument: T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c for earlier term 2. Current term is 3.
I20260925 03:04:19.018775 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c23337a7cb74184a0e57b7095cb6a11" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:19.022881 24473 raft_consensus.cc:2454] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 2.
I20260925 03:04:18.965078 25485 raft_consensus.cc:482] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:19.024192 24436 leader_election.cc:302] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:19.023928 25485 raft_consensus.cc:504] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:19.025130 24436 leader_election.cc:302] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:19.026140 25485 leader_election.cc:288] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:19.027254 25641 raft_consensus.cc:2781] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260925 03:04:19.027437 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5aec38e66a947c681e6c2a72b05e6ca" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:18.951833 25635 raft_consensus.cc:2781] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 03:04:19.028065 25641 raft_consensus.cc:482] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:19.028220 25635 raft_consensus.cc:686] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 LEADER]: Becoming Leader. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:04:19.019798 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f1fac028a144058b92331c0d9b44ebf" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:19.029858 24468 raft_consensus.cc:2454] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 2.
I20260925 03:04:19.016556 24511 leader_election.cc:302] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:19.015025 25561 raft_consensus.cc:504] T c85f5d1d3f004cf5817bb594fd8e6469 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:18.951418 25642 raft_consensus.cc:686] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 LEADER]: Becoming Leader. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:04:19.031359 24588 leader_election.cc:302] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:19.042194 25529 leader_election.cc:288] T c88cbd39e01f44628a094af516be2833 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:18.978536 25640 leader_election.cc:288] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:19.042979 25640 raft_consensus.cc:2781] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260925 03:04:19.031749 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35992316cb8d4c028333388eff02ecde" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:19.032824 24616 raft_consensus.cc:2454] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 2.
I20260925 03:04:19.043347 25640 raft_consensus.cc:482] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:19.044857 25560 leader_election.cc:288] T b030d84c670b48d5bff337ccc0c77626 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:18.976236 25516 raft_consensus.cc:504] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:19.045714 25640 raft_consensus.cc:3037] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:19.041122 25622 raft_consensus.cc:482] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:19.028544 25641 raft_consensus.cc:3037] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:19.041939 25561 leader_election.cc:288] T c85f5d1d3f004cf5817bb594fd8e6469 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:19.031843 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c81a1b3b616492ab6c964f45af36077" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:19.044338 25642 consensus_queue.cc:237] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:19.043533 24470 raft_consensus.cc:2454] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 2.
I20260925 03:04:19.026386 25564 raft_consensus.cc:2781] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260925 03:04:19.048278 24546 raft_consensus.cc:2454] T b030d84c670b48d5bff337ccc0c77626 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 1.
I20260925 03:04:19.026803 25485 raft_consensus.cc:2781] T 442d4a29b1d743599f4b6b4871e26668 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260925 03:04:19.048904 24369 catalog_manager.cc:5495] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "869b0a8575b54d08ae2817fad5dd42d5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260925 03:04:18.973410 25557 raft_consensus.cc:482] T 37a157996e944db2906110d87ecb0c54 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:18.970217 25563 leader_election.cc:288] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:18.981042 25639 raft_consensus.cc:482] T 74443a0a1dc34d88a2f42e3b722db405 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting leader election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:19.017102 25551 raft_consensus.cc:482] T 06f4a7eaf6534e4986c9fd35911038e4 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:18.986372 25645 raft_consensus.cc:3037] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:19.053166 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:19.056846 25564 raft_consensus.cc:482] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:19.065104 25564 raft_consensus.cc:3037] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:19.066138 25637 raft_consensus.cc:504] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:19.058519 24465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37a157996e944db2906110d87ecb0c54" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:19.068100 25637 leader_election.cc:288] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:19.052562 25622 raft_consensus.cc:504] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:19.058350 24615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c85f5d1d3f004cf5817bb594fd8e6469" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:18.968154 25602 leader_election.cc:288] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:19.057686 24436 leader_election.cc:302] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:19.051846 24468 raft_consensus.cc:2454] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 2.
I20260925 03:04:19.049080 25638 raft_consensus.cc:482] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:19.070196 25199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c81a1b3b616492ab6c964f45af36077" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:19.096282 25199 raft_consensus.cc:2454] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 2.
I20260925 03:04:19.071619 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ac285175464fbda67211d9574db415" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:19.096952 24472 raft_consensus.cc:2454] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 2.
I20260925 03:04:19.097048 24588 leader_election.cc:302] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:19.077050 24615 raft_consensus.cc:3037] T c85f5d1d3f004cf5817bb594fd8e6469 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:19.069099 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e725b82985746b39c0cc8f671c95353" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:19.098721 24464 raft_consensus.cc:2379] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c in current term 2: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:04:19.098671 24588 leader_election.cc:302] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:19.051232 25547 leader_election.cc:288] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 2 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:19.057914 25485 raft_consensus.cc:482] T 442d4a29b1d743599f4b6b4871e26668 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:19.115163 24473 raft_consensus.cc:2454] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 2.
I20260925 03:04:19.047938 25533 raft_consensus.cc:504] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:19.061054 25584 raft_consensus.cc:482] T 37a157996e944db2906110d87ecb0c54 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:19.059439 25516 leader_election.cc:288] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:19.109952 25637 raft_consensus.cc:2781] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260925 03:04:19.116014 25563 raft_consensus.cc:2781] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260925 03:04:19.064154 25560 raft_consensus.cc:482] T f22ddd7d1c444dc58f45a1fb0b3e02bc P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:19.107553 24469 raft_consensus.cc:2454] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 2.
I20260925 03:04:19.116065 25485 raft_consensus.cc:3037] T 442d4a29b1d743599f4b6b4871e26668 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:19.069087 25648 raft_consensus.cc:2781] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260925 03:04:19.021845 25540 consensus_queue.cc:237] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:19.144506 25505 raft_consensus.cc:504] T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:19.146579 25622 leader_election.cc:288] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:19.132499 24615 raft_consensus.cc:2454] T c85f5d1d3f004cf5817bb594fd8e6469 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 3.
I20260925 03:04:19.107414 25638 raft_consensus.cc:504] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:19.069253 25557 raft_consensus.cc:504] T 37a157996e944db2906110d87ecb0c54 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:19.131640 25637 raft_consensus.cc:482] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:19.134562 25563 raft_consensus.cc:482] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:19.140834 25560 raft_consensus.cc:504] T f22ddd7d1c444dc58f45a1fb0b3e02bc P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:19.142338 24588 leader_election.cc:302] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:19.142901 25533 leader_election.cc:288] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:19.144124 25535 raft_consensus.cc:504] T 0b5df303cd6a4d59aca5c18bbbff24ed P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:19.144769 25648 raft_consensus.cc:482] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:19.182188 25648 raft_consensus.cc:3037] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:19.075156 25639 raft_consensus.cc:3037] T 74443a0a1dc34d88a2f42e3b722db405 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Advancing to term 2
W20260925 03:04:19.154886 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 212424us
I20260925 03:04:19.093117 25551 raft_consensus.cc:504] T 06f4a7eaf6534e4986c9fd35911038e4 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:19.189075 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:19.118814 25604 raft_consensus.cc:2781] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260925 03:04:19.189827 24473 raft_consensus.cc:2379] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 2: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:04:19.190068 25604 raft_consensus.cc:482] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:19.190439 25604 raft_consensus.cc:3037] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:19.190703 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c88cbd39e01f44628a094af516be2833" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:19.191246 24471 raft_consensus.cc:3037] T c88cbd39e01f44628a094af516be2833 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:19.191743 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b627ecb4a5384ef392aa695e06599c39" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:19.192286 24472 raft_consensus.cc:3037] T b627ecb4a5384ef392aa695e06599c39 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:19.192871 25533 raft_consensus.cc:2781] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260925 03:04:19.193253 25533 raft_consensus.cc:482] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:19.193655 25533 raft_consensus.cc:3037] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:19.153532 25485 raft_consensus.cc:504] T 442d4a29b1d743599f4b6b4871e26668 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:19.146775 25505 leader_election.cc:288] T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:19.116257 24616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5aec38e66a947c681e6c2a72b05e6ca" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
W20260925 03:04:19.187547 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 45311.3us
I20260925 03:04:19.197556 24616 raft_consensus.cc:2379] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1bc1f2e62ab44af995165bd5116a9d43 in current term 2: Already voted for candidate eeab93abfdb141a895002ec9f312296c in this term.
I20260925 03:04:19.140163 25539 raft_consensus.cc:482] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:19.163354 25637 raft_consensus.cc:3037] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:19.173856 25563 raft_consensus.cc:3037] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:19.130153 25635 consensus_queue.cc:237] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:19.209344 25539 raft_consensus.cc:504] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:19.199642 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b030d84c670b48d5bff337ccc0c77626" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:19.191332 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b030d84c670b48d5bff337ccc0c77626" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:19.208511 24472 raft_consensus.cc:2454] T b627ecb4a5384ef392aa695e06599c39 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 3.
I20260925 03:04:19.200444 24471 raft_consensus.cc:2454] T c88cbd39e01f44628a094af516be2833 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 3.
I20260925 03:04:19.199343 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06f4a7eaf6534e4986c9fd35911038e4" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:19.222883 24473 raft_consensus.cc:2379] T 06f4a7eaf6534e4986c9fd35911038e4 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c in current term 2: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:04:19.196425 25638 leader_election.cc:288] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:19.198086 25485 leader_election.cc:288] T 442d4a29b1d743599f4b6b4871e26668 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:19.221652 24514 leader_election.cc:302] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:04:19.226855 24514 leader_election.cc:302] T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:04:19.229741 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15861cd28e354598a3c6bfc4337c34bb" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:19.136652 25584 raft_consensus.cc:504] T 37a157996e944db2906110d87ecb0c54 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:19.189750 25551 leader_election.cc:288] T 06f4a7eaf6534e4986c9fd35911038e4 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:19.210793 25505 raft_consensus.cc:482] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:19.198282 24613 raft_consensus.cc:2454] T 37a157996e944db2906110d87ecb0c54 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 1.
I20260925 03:04:19.220932 25643 raft_consensus.cc:482] T c85f5d1d3f004cf5817bb594fd8e6469 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:19.241274 25557 leader_election.cc:288] T 37a157996e944db2906110d87ecb0c54 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:19.241344 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b5df303cd6a4d59aca5c18bbbff24ed" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:19.243535 24472 raft_consensus.cc:2379] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c in current term 2: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:04:19.239660 25535 leader_election.cc:288] T 0b5df303cd6a4d59aca5c18bbbff24ed P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:19.239744 25539 leader_election.cc:288] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:19.242216 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37a157996e944db2906110d87ecb0c54" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:19.241554 25643 raft_consensus.cc:504] T c85f5d1d3f004cf5817bb594fd8e6469 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:19.249531 25560 leader_election.cc:288] T f22ddd7d1c444dc58f45a1fb0b3e02bc P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:19.250389 25560 raft_consensus.cc:2781] T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 03:04:19.250932 25560 raft_consensus.cc:686] T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 LEADER]: Becoming Leader. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:04:19.244937 24465 raft_consensus.cc:2454] T 37a157996e944db2906110d87ecb0c54 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 2.
I20260925 03:04:19.234006 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35992316cb8d4c028333388eff02ecde" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:19.253489 24469 raft_consensus.cc:2379] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c in current term 2: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:04:19.231320 25555 raft_consensus.cc:504] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:19.256395 25555 leader_election.cc:288] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:19.256945 25533 raft_consensus.cc:504] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:19.240350 25584 leader_election.cc:288] T 37a157996e944db2906110d87ecb0c54 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:19.237484 24618 raft_consensus.cc:2454] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 3.
I20260925 03:04:19.237663 25516 raft_consensus.cc:2781] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260925 03:04:19.263793 25535 raft_consensus.cc:482] T 11bb46bce3564841b7674e6a693bd2e6 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:19.251762 25648 raft_consensus.cc:504] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:19.258354 25564 raft_consensus.cc:504] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:19.235688 25529 consensus_queue.cc:1035] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.026s
I20260925 03:04:19.258635 25539 raft_consensus.cc:482] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:19.270880 25648 leader_election.cc:288] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
W20260925 03:04:19.276675 24511 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 260147us
I20260925 03:04:19.252089 25643 leader_election.cc:288] T c85f5d1d3f004cf5817bb594fd8e6469 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:19.264894 25516 raft_consensus.cc:482] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:19.247423 25505 raft_consensus.cc:504] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:19.251876 25560 consensus_queue.cc:237] T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:19.283717 25516 raft_consensus.cc:3037] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:19.289608 24511 leader_election.cc:302] T 37a157996e944db2906110d87ecb0c54 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
W20260925 03:04:19.297297 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 343631us
I20260925 03:04:19.272320 25535 raft_consensus.cc:504] T 11bb46bce3564841b7674e6a693bd2e6 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:19.301553 24615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c88cbd39e01f44628a094af516be2833" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:19.299077 24586 leader_election.cc:302] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:04:19.307621 24618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:19.308151 25650 raft_consensus.cc:2781] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Leader election won for term 2
I20260925 03:04:19.308528 24618 raft_consensus.cc:2354] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 2. Current term is 3.
I20260925 03:04:19.308790 25650 raft_consensus.cc:686] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [term 2 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:19.309855 25650 consensus_queue.cc:237] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:19.312265 24618 raft_consensus.cc:1261] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260925 03:04:19.314406 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442d4a29b1d743599f4b6b4871e26668" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:19.315223 24619 raft_consensus.cc:3037] T 442d4a29b1d743599f4b6b4871e26668 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Advancing to term 3
I20260925 03:04:19.320581 24586 leader_election.cc:302] T 442d4a29b1d743599f4b6b4871e26668 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:04:19.324141 24586 leader_election.cc:302] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:04:19.279026 25539 raft_consensus.cc:504] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:19.330729 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b030d84c670b48d5bff337ccc0c77626" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:19.331552 24620 raft_consensus.cc:2379] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 2: Already voted for candidate eeab93abfdb141a895002ec9f312296c in this term.
I20260925 03:04:19.299538 25648 raft_consensus.cc:2781] T 37a157996e944db2906110d87ecb0c54 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260925 03:04:19.284189 25604 raft_consensus.cc:504] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:19.323911 24619 raft_consensus.cc:2454] T 442d4a29b1d743599f4b6b4871e26668 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 3.
I20260925 03:04:19.299037 25564 leader_election.cc:288] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:19.340649 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f22ddd7d1c444dc58f45a1fb0b3e02bc" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:19.295979 24616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71f461a6e2ff434993b78a2d077f0abd" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:19.301630 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:19.341733 24620 raft_consensus.cc:2454] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 2.
W20260925 03:04:19.351043 24436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 293425us
I20260925 03:04:19.356346 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c85f5d1d3f004cf5817bb594fd8e6469" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:19.357069 24543 raft_consensus.cc:3037] T c85f5d1d3f004cf5817bb594fd8e6469 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Advancing to term 3
I20260925 03:04:19.333894 25572 raft_consensus.cc:2781] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Leader election won for term 2
I20260925 03:04:19.358955 24468 raft_consensus.cc:2379] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c in current term 3: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:04:19.365320 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37a157996e944db2906110d87ecb0c54" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:19.366004 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60d45b7d467d4e1ab92367f6d7d52b15" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:19.366894 24511 leader_election.cc:398] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Vote denied by peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105) with higher term. Message: Invalid argument: T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 2. Current term is 3.
I20260925 03:04:19.360121 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06f4a7eaf6534e4986c9fd35911038e4" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:19.290537 25649 consensus_queue.cc:1035] T 11bb46bce3564841b7674e6a693bd2e6 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.074s
I20260925 03:04:19.360582 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35992316cb8d4c028333388eff02ecde" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:19.367729 24542 raft_consensus.cc:2454] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 2.
I20260925 03:04:19.367785 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06f4a7eaf6534e4986c9fd35911038e4" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:19.368170 24617 raft_consensus.cc:3037] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:19.368675 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f1fac028a144058b92331c0d9b44ebf" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:19.369341 24542 raft_consensus.cc:2454] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 2.
I20260925 03:04:19.370167 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e725b82985746b39c0cc8f671c95353" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:19.361135 24618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f1fac028a144058b92331c0d9b44ebf" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:19.362701 25604 leader_election.cc:288] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:19.341300 25648 raft_consensus.cc:482] T 37a157996e944db2906110d87ecb0c54 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:19.348590 25636 raft_consensus.cc:482] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:19.338404 25647 raft_consensus.cc:2741] T 442d4a29b1d743599f4b6b4871e26668 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20260925 03:04:19.369496 24615 raft_consensus.cc:2454] T c88cbd39e01f44628a094af516be2833 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 3.
I20260925 03:04:19.348405 25558 raft_consensus.cc:482] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:19.365645 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15861cd28e354598a3c6bfc4337c34bb" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:19.385529 25533 leader_election.cc:288] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 3 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:19.400102 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c85f5d1d3f004cf5817bb594fd8e6469" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:19.401017 24619 raft_consensus.cc:2354] T c85f5d1d3f004cf5817bb594fd8e6469 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 2. Current term is 3.
I20260925 03:04:19.388427 25505 leader_election.cc:288] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:19.388350 24543 raft_consensus.cc:2454] T c85f5d1d3f004cf5817bb594fd8e6469 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 3.
I20260925 03:04:19.394317 24547 raft_consensus.cc:2454] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 2.
I20260925 03:04:19.353884 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:19.367211 24511 leader_election.cc:401] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
W20260925 03:04:19.618232 24511 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 251314us
I20260925 03:04:19.619087 25649 raft_consensus.cc:3037] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Advancing to term 3
I20260925 03:04:19.498056 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ac285175464fbda67211d9574db415" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:19.399453 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b5df303cd6a4d59aca5c18bbbff24ed" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
W20260925 03:04:19.619890 25663 log.cc:924] Time spent T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.171s user 0.003s sys 0.009s
I20260925 03:04:19.621328 25664 consensus_queue.cc:1035] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260925 03:04:19.358276 24616 raft_consensus.cc:2454] T 71f461a6e2ff434993b78a2d077f0abd P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 1.
I20260925 03:04:19.624393 24511 leader_election.cc:302] T 442d4a29b1d743599f4b6b4871e26668 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:19.423975 25639 raft_consensus.cc:504] T 74443a0a1dc34d88a2f42e3b722db405 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:19.625744 25664 raft_consensus.cc:2781] T 442d4a29b1d743599f4b6b4871e26668 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 03:04:19.628208 25639 leader_election.cc:288] T 74443a0a1dc34d88a2f42e3b722db405 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:19.428476 24617 raft_consensus.cc:2454] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 3.
I20260925 03:04:19.411612 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b627ecb4a5384ef392aa695e06599c39" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:19.397814 25196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71f461a6e2ff434993b78a2d077f0abd" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:19.585186 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:19.628818 25664 raft_consensus.cc:686] T 442d4a29b1d743599f4b6b4871e26668 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 LEADER]: Becoming Leader. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:04:19.630707 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bb46bce3564841b7674e6a693bd2e6" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:19.587841 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35992316cb8d4c028333388eff02ecde" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:19.356457 24469 raft_consensus.cc:3037] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:19.408546 25535 leader_election.cc:288] T 11bb46bce3564841b7674e6a693bd2e6 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:19.400274 25558 raft_consensus.cc:504] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:19.610265 25517 raft_consensus.cc:482] T 71f461a6e2ff434993b78a2d077f0abd P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:19.635411 24621 raft_consensus.cc:3037] T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:19.635461 24546 raft_consensus.cc:2354] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c for earlier term 2. Current term is 3.
I20260925 03:04:19.636112 25517 raft_consensus.cc:504] T 71f461a6e2ff434993b78a2d077f0abd P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:19.639163 25517 leader_election.cc:288] T 71f461a6e2ff434993b78a2d077f0abd P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:19.438068 25539 leader_election.cc:288] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:19.437059 25572 raft_consensus.cc:686] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [term 2 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:19.640192 25196 raft_consensus.cc:2454] T 71f461a6e2ff434993b78a2d077f0abd P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 2.
I20260925 03:04:19.643955 25558 leader_election.cc:288] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:19.610968 25516 raft_consensus.cc:504] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:19.645745 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b030d84c670b48d5bff337ccc0c77626" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:19.646505 24543 raft_consensus.cc:2454] T b030d84c670b48d5bff337ccc0c77626 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 1.
I20260925 03:04:19.646975 24548 raft_consensus.cc:2354] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c for earlier term 2. Current term is 3.
I20260925 03:04:19.647138 25516 leader_election.cc:288] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:19.650512 25199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5aec38e66a947c681e6c2a72b05e6ca" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:19.651510 25199 raft_consensus.cc:2454] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 2.
I20260925 03:04:19.654817 25165 leader_election.cc:302] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:04:19.653301 25196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43"
I20260925 03:04:19.657577 25196 raft_consensus.cc:3037] T 29637f5a4a504ce6ba29033ccd0cd716 P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 LEARNER]: Advancing to term 3
I20260925 03:04:19.658380 25517 raft_consensus.cc:2781] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 03:04:19.609565 24586 leader_election.cc:302] T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:04:19.407567 24542 raft_consensus.cc:2454] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 2.
I20260925 03:04:19.396239 25636 raft_consensus.cc:504] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:19.388624 25637 raft_consensus.cc:504] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:19.387532 25640 raft_consensus.cc:504] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:19.663697 25198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15861cd28e354598a3c6bfc4337c34bb" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:19.389160 25641 raft_consensus.cc:504] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:19.665406 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bb46bce3564841b7674e6a693bd2e6" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:19.665609 25637 leader_election.cc:288] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 3 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:19.666461 24470 raft_consensus.cc:2454] T 11bb46bce3564841b7674e6a693bd2e6 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 2.
I20260925 03:04:19.666829 25641 leader_election.cc:288] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:19.357632 25645 raft_consensus.cc:504] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:19.668530 25584 raft_consensus.cc:482] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:19.669449 24539 raft_consensus.cc:2454] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 2.
I20260925 03:04:19.669198 25584 raft_consensus.cc:504] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:19.380493 25648 raft_consensus.cc:3037] T 37a157996e944db2906110d87ecb0c54 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:19.671758 25584 leader_election.cc:288] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:19.641530 25572 consensus_queue.cc:237] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:04:19.501427 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 274512us
I20260925 03:04:19.626652 24622 raft_consensus.cc:3037] T 74443a0a1dc34d88a2f42e3b722db405 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:19.607115 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37a157996e944db2906110d87ecb0c54" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:19.626564 24511 leader_election.cc:302] T f22ddd7d1c444dc58f45a1fb0b3e02bc P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:19.664516 25198 raft_consensus.cc:2454] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 2.
I20260925 03:04:19.677143 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5aec38e66a947c681e6c2a72b05e6ca" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:19.673439 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71f461a6e2ff434993b78a2d077f0abd" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:19.678277 24468 raft_consensus.cc:2454] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 2.
I20260925 03:04:19.678591 24617 raft_consensus.cc:2454] T 71f461a6e2ff434993b78a2d077f0abd P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 1.
W20260925 03:04:19.682147 24511 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 55841.8us
I20260925 03:04:19.683257 24511 leader_election.cc:302] T c88cbd39e01f44628a094af516be2833 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:19.683923 25516 raft_consensus.cc:2781] T f22ddd7d1c444dc58f45a1fb0b3e02bc P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 03:04:19.684887 25516 raft_consensus.cc:482] T f22ddd7d1c444dc58f45a1fb0b3e02bc P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:19.685230 25640 leader_election.cc:288] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 3 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:19.685464 25516 raft_consensus.cc:3037] T f22ddd7d1c444dc58f45a1fb0b3e02bc P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:19.677398 24514 leader_election.cc:302] T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:04:19.669828 25645 leader_election.cc:288] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:19.678438 24618 raft_consensus.cc:2354] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 2. Current term is 3.
I20260925 03:04:19.689533 25560 raft_consensus.cc:2781] T c88cbd39e01f44628a094af516be2833 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 03:04:19.690534 25198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c81a1b3b616492ab6c964f45af36077" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43"
I20260925 03:04:19.691305 25198 raft_consensus.cc:3037] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:19.662068 25517 raft_consensus.cc:686] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 LEADER]: Becoming Leader. State: Replica: 1bc1f2e62ab44af995165bd5116a9d43, State: Running, Role: LEADER
I20260925 03:04:19.693193 25560 raft_consensus.cc:686] T c88cbd39e01f44628a094af516be2833 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 LEADER]: Becoming Leader. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:04:19.693428 25643 raft_consensus.cc:2781] T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 03:04:19.693922 25643 raft_consensus.cc:482] T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
W20260925 03:04:19.692492 25668 log.cc:924] Time spent T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.052s user 0.002s sys 0.000s
I20260925 03:04:19.694490 25643 raft_consensus.cc:3037] T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:19.701128 24514 leader_election.cc:302] T b030d84c670b48d5bff337ccc0c77626 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:19.694481 25560 consensus_queue.cc:237] T c88cbd39e01f44628a094af516be2833 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:19.701510 24622 raft_consensus.cc:2454] T 74443a0a1dc34d88a2f42e3b722db405 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 3.
I20260925 03:04:19.700186 25648 raft_consensus.cc:504] T 37a157996e944db2906110d87ecb0c54 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:19.700704 25664 consensus_queue.cc:237] T 442d4a29b1d743599f4b6b4871e26668 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:19.708005 25584 raft_consensus.cc:482] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:19.708137 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "420163a6cdce412ebc21c25fe3835638" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:19.709178 24541 raft_consensus.cc:3037] T 420163a6cdce412ebc21c25fe3835638 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Advancing to term 3
I20260925 03:04:19.703552 24585 leader_election.cc:302] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:19.710959 25584 raft_consensus.cc:504] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:19.712289 24621 raft_consensus.cc:2454] T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 3.
I20260925 03:04:19.712644 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74443a0a1dc34d88a2f42e3b722db405" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:19.714236 25657 raft_consensus.cc:3037] T b030d84c670b48d5bff337ccc0c77626 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:19.714375 24544 raft_consensus.cc:2379] T 37a157996e944db2906110d87ecb0c54 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 in current term 3: Already voted for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in this term.
I20260925 03:04:19.720721 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f1fac028a144058b92331c0d9b44ebf" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:19.720940 25666 raft_consensus.cc:2781] T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Leader election won for term 2
I20260925 03:04:19.721501 24470 raft_consensus.cc:3037] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:19.721563 25666 raft_consensus.cc:686] T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c [term 2 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:19.721801 24465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c81a1b3b616492ab6c964f45af36077" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:19.722561 24465 raft_consensus.cc:3037] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:19.722726 25666 consensus_queue.cc:237] T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:19.727911 25649 raft_consensus.cc:2726] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105) with higher term. Message: Invalid argument: T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 2. Current term is 3.
I20260925 03:04:19.728804 24540 raft_consensus.cc:2454] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 3.
I20260925 03:04:19.728965 24468 raft_consensus.cc:2354] T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c for earlier term 2. Current term is 3.
I20260925 03:04:19.732616 25648 leader_election.cc:288] T 37a157996e944db2906110d87ecb0c54 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:19.737174 24470 raft_consensus.cc:2454] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 3.
I20260925 03:04:19.693846 25517 consensus_queue.cc:237] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:19.745579 25198 raft_consensus.cc:2454] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 3.
I20260925 03:04:19.746135 24511 leader_election.cc:398] T c85f5d1d3f004cf5817bb594fd8e6469 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Vote denied by peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105) with higher term. Message: Invalid argument: T c85f5d1d3f004cf5817bb594fd8e6469 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 2. Current term is 3.
I20260925 03:04:19.749161 24511 leader_election.cc:401] T c85f5d1d3f004cf5817bb594fd8e6469 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
W20260925 03:04:19.753999 24585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.4:43979, user_credentials={real_user=slave}} blocked reactor thread for 80544.9us
W20260925 03:04:19.755803 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 54752.6us
I20260925 03:04:19.761454 24368 catalog_manager.cc:5495] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: UNKNOWN } } }
I20260925 03:04:19.765106 25572 raft_consensus.cc:2781] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260925 03:04:19.765669 25572 raft_consensus.cc:482] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:19.766148 25572 raft_consensus.cc:3037] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:19.766655 24514 leader_election.cc:302] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:04:19.767488 25636 leader_election.cc:288] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:19.763705 24368 catalog_manager.cc:5495] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: UNKNOWN } } }
I20260925 03:04:19.762967 24371 catalog_manager.cc:5495] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 reported cstate change: term changed from 1 to 3, now has a pending config: VOTER 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1), VOTER a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2), VOTER eeab93abfdb141a895002ec9f312296c (127.22.193.3), VOTER 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4). New cstate: current_term: 3 leader_uuid: "869b0a8575b54d08ae2817fad5dd42d5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } pending_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:19.780825 24588 leader_election.cc:302] T 11bb46bce3564841b7674e6a693bd2e6 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:19.782351 25640 raft_consensus.cc:2781] T 11bb46bce3564841b7674e6a693bd2e6 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260925 03:04:19.782779 25640 raft_consensus.cc:482] T 11bb46bce3564841b7674e6a693bd2e6 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:19.783133 25640 raft_consensus.cc:3037] T 11bb46bce3564841b7674e6a693bd2e6 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:19.785189 24588 leader_election.cc:398] T 74443a0a1dc34d88a2f42e3b722db405 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 election: Vote denied by peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187) with higher term. Message: Invalid argument: T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c for earlier term 2. Current term is 3.
I20260925 03:04:19.759228 25584 leader_election.cc:288] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:19.785580 24588 leader_election.cc:401] T 74443a0a1dc34d88a2f42e3b722db405 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260925 03:04:19.794917 25555 raft_consensus.cc:482] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
W20260925 03:04:19.795711 25563 consensus_meta.cc:287] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c: Time spent flushing consensus metadata: real 0.565s user 0.009s sys 0.009s
I20260925 03:04:19.803172 25563 raft_consensus.cc:504] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:19.804646 25666 raft_consensus.cc:2726] T 74443a0a1dc34d88a2f42e3b722db405 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187) with higher term. Message: Invalid argument: T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c for earlier term 2. Current term is 3.
I20260925 03:04:19.805393 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c85f5d1d3f004cf5817bb594fd8e6469" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:19.806087 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35992316cb8d4c028333388eff02ecde" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:19.806406 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442d4a29b1d743599f4b6b4871e26668" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:19.806841 24473 raft_consensus.cc:3037] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:19.807331 24618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06f4a7eaf6534e4986c9fd35911038e4" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:19.807588 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f1fac028a144058b92331c0d9b44ebf" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:19.808343 24464 raft_consensus.cc:2354] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 2. Current term is 3.
I20260925 03:04:19.795722 25555 raft_consensus.cc:504] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:19.807334 24468 raft_consensus.cc:3037] T 442d4a29b1d743599f4b6b4871e26668 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:19.806938 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f22ddd7d1c444dc58f45a1fb0b3e02bc" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:19.809563 24472 raft_consensus.cc:2379] T f22ddd7d1c444dc58f45a1fb0b3e02bc P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 2: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:04:19.810370 25555 leader_election.cc:288] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:19.812605 25563 leader_election.cc:288] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 3 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:19.756999 25643 raft_consensus.cc:504] T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:19.798866 25649 raft_consensus.cc:2781] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 03:04:19.802564 24465 raft_consensus.cc:2454] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 3.
I20260925 03:04:19.806392 24471 raft_consensus.cc:2354] T c85f5d1d3f004cf5817bb594fd8e6469 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 2. Current term is 3.
I20260925 03:04:19.815263 24472 raft_consensus.cc:1261] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260925 03:04:19.817378 24468 raft_consensus.cc:2454] T 442d4a29b1d743599f4b6b4871e26668 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 3.
I20260925 03:04:19.796977 24588 leader_election.cc:302] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:19.818467 24473 raft_consensus.cc:2454] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 3.
I20260925 03:04:19.822942 24618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37a157996e944db2906110d87ecb0c54" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:19.823206 24467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60d45b7d467d4e1ab92367f6d7d52b15" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:19.792732 25657 raft_consensus.cc:2726] T b030d84c670b48d5bff337ccc0c77626 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260925 03:04:19.828907 24511 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 115634us
I20260925 03:04:19.830562 24470 raft_consensus.cc:1261] T 11bb46bce3564841b7674e6a693bd2e6 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260925 03:04:19.832026 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f22ddd7d1c444dc58f45a1fb0b3e02bc" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:19.834669 25664 raft_consensus.cc:2726] T c85f5d1d3f004cf5817bb594fd8e6469 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105) with higher term. Message: Invalid argument: T c85f5d1d3f004cf5817bb594fd8e6469 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 2. Current term is 3.
I20260925 03:04:19.834723 24468 raft_consensus.cc:2454] T f22ddd7d1c444dc58f45a1fb0b3e02bc P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 2.
I20260925 03:04:19.837684 24588 leader_election.cc:302] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:19.837078 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ac285175464fbda67211d9574db415" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:19.839470 24467 raft_consensus.cc:3037] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:19.839764 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71f461a6e2ff434993b78a2d077f0abd" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:19.827953 25572 raft_consensus.cc:504] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:19.840551 24540 raft_consensus.cc:2454] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 2.
I20260925 03:04:19.841722 24471 raft_consensus.cc:1261] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260925 03:04:19.842137 25572 leader_election.cc:288] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 3 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:19.831729 25649 raft_consensus.cc:686] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 LEADER]: Becoming Leader. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
W20260925 03:04:19.842711 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 233775us
I20260925 03:04:19.844842 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ac285175464fbda67211d9574db415" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:19.847287 24545 raft_consensus.cc:2454] T 0b5df303cd6a4d59aca5c18bbbff24ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 2.
I20260925 03:04:19.848080 25572 consensus_queue.cc:1035] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260925 03:04:19.848276 25649 consensus_queue.cc:237] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:19.848563 25163 leader_election.cc:302] T 71f461a6e2ff434993b78a2d077f0abd P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:04:19.839192 25666 raft_consensus.cc:2781] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader election won for term 3
I20260925 03:04:19.850472 24588 leader_election.cc:302] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:19.851311 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37a157996e944db2906110d87ecb0c54" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:19.851961 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5aec38e66a947c681e6c2a72b05e6ca" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:19.853398 24472 raft_consensus.cc:3037] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:19.854053 24471 raft_consensus.cc:3037] T 37a157996e944db2906110d87ecb0c54 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:19.855243 25199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5aec38e66a947c681e6c2a72b05e6ca" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43"
I20260925 03:04:19.860486 25622 raft_consensus.cc:2781] T 71f461a6e2ff434993b78a2d077f0abd P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260925 03:04:19.860642 25643 leader_election.cc:288] T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:19.860962 25622 raft_consensus.cc:482] T 71f461a6e2ff434993b78a2d077f0abd P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:19.861393 25622 raft_consensus.cc:3037] T 71f461a6e2ff434993b78a2d077f0abd P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:19.862987 24511 leader_election.cc:302] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:19.864430 25648 raft_consensus.cc:2781] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260925 03:04:19.864827 25648 raft_consensus.cc:482] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:19.865170 25648 raft_consensus.cc:3037] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:19.865310 25563 raft_consensus.cc:2781] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260925 03:04:19.866098 25563 raft_consensus.cc:482] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:19.866786 25563 raft_consensus.cc:3037] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:19.869536 24511 leader_election.cc:302] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:19.871033 25643 raft_consensus.cc:2781] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 03:04:19.872198 25643 raft_consensus.cc:686] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 LEADER]: Becoming Leader. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:04:19.876377 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15861cd28e354598a3c6bfc4337c34bb" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:19.877624 24466 raft_consensus.cc:2454] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 2.
I20260925 03:04:19.882009 25640 raft_consensus.cc:504] T 11bb46bce3564841b7674e6a693bd2e6 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:19.882923 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f1fac028a144058b92331c0d9b44ebf" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:19.883780 24544 raft_consensus.cc:3037] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:19.838496 24618 raft_consensus.cc:2454] T 37a157996e944db2906110d87ecb0c54 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 1.
I20260925 03:04:19.885586 25666 raft_consensus.cc:686] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c [term 3 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:19.888968 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74443a0a1dc34d88a2f42e3b722db405" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:19.893805 25666 consensus_queue.cc:237] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:19.896445 25196 raft_consensus.cc:2454] T 29637f5a4a504ce6ba29033ccd0cd716 P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 LEARNER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 3.
I20260925 03:04:19.874073 25643 consensus_queue.cc:237] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:19.899283 25622 raft_consensus.cc:504] T 71f461a6e2ff434993b78a2d077f0abd P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:19.903523 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ac285175464fbda67211d9574db415" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:19.904341 24540 raft_consensus.cc:3037] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:19.907084 25636 raft_consensus.cc:2781] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader election won for term 3
I20260925 03:04:19.907665 25636 raft_consensus.cc:686] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c [term 3 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:19.908680 25636 consensus_queue.cc:237] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:19.914518 24544 raft_consensus.cc:2454] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 3.
I20260925 03:04:19.916213 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f22ddd7d1c444dc58f45a1fb0b3e02bc" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:19.916427 24467 raft_consensus.cc:2454] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 3.
I20260925 03:04:19.904253 24514 leader_election.cc:398] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 election: Vote denied by peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187) with higher term. Message: Invalid argument: T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 2. Current term is 3.
I20260925 03:04:19.918581 24514 leader_election.cc:401] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260925 03:04:19.919088 24511 leader_election.cc:398] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 election: Vote denied by peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105) with higher term. Message: Invalid argument: T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 2. Current term is 3.
W20260925 03:04:19.916090 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 65677.8us
W20260925 03:04:19.922176 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 47554.8us
I20260925 03:04:19.921713 24464 raft_consensus.cc:2454] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 2.
I20260925 03:04:19.924226 24471 raft_consensus.cc:2454] T 37a157996e944db2906110d87ecb0c54 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 3.
I20260925 03:04:19.924984 24472 raft_consensus.cc:2454] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 3.
I20260925 03:04:19.907860 25516 raft_consensus.cc:504] T f22ddd7d1c444dc58f45a1fb0b3e02bc P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:19.927055 24514 leader_election.cc:398] T c85f5d1d3f004cf5817bb594fd8e6469 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187) with higher term. Message: Invalid argument: T c85f5d1d3f004cf5817bb594fd8e6469 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 2. Current term is 3.
I20260925 03:04:19.929580 25645 raft_consensus.cc:2726] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187) with higher term. Message: Invalid argument: T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 2. Current term is 3.
I20260925 03:04:19.931663 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15861cd28e354598a3c6bfc4337c34bb" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:19.927860 24588 leader_election.cc:302] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters: 1bc1f2e62ab44af995165bd5116a9d43
I20260925 03:04:19.934306 24468 raft_consensus.cc:2454] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 2.
I20260925 03:04:19.935945 25637 raft_consensus.cc:2781] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader election won for term 3
I20260925 03:04:19.936523 25637 raft_consensus.cc:686] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [term 3 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:19.937151 25622 leader_election.cc:288] T 71f461a6e2ff434993b78a2d077f0abd P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 3 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:19.937551 25637 consensus_queue.cc:237] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:19.937952 25664 consensus_queue.cc:1035] T 11bb46bce3564841b7674e6a693bd2e6 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260925 03:04:19.941828 24514 leader_election.cc:302] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:04:19.944813 24514 leader_election.cc:302] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:04:19.945899 24545 raft_consensus.cc:2454] T f22ddd7d1c444dc58f45a1fb0b3e02bc P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 2.
I20260925 03:04:19.948339 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b030d84c670b48d5bff337ccc0c77626" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:19.949028 24514 leader_election.cc:302] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:04:19.951838 24586 leader_election.cc:398] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 election: Vote denied by peer a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389) with higher term. Message: Invalid argument: T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c for earlier term 2. Current term is 3.
I20260925 03:04:19.952121 24586 leader_election.cc:401] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260925 03:04:19.954470 24617 raft_consensus.cc:3037] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:19.954523 25555 raft_consensus.cc:2781] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 03:04:19.953020 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "420163a6cdce412ebc21c25fe3835638" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:19.945343 25657 raft_consensus.cc:2781] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 03:04:19.939793 25645 consensus_queue.cc:1035] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.009s
I20260925 03:04:19.966421 24436 leader_election.cc:302] T c85f5d1d3f004cf5817bb594fd8e6469 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:19.968570 25558 raft_consensus.cc:2781] T c85f5d1d3f004cf5817bb594fd8e6469 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 03:04:19.969221 25558 raft_consensus.cc:686] T c85f5d1d3f004cf5817bb594fd8e6469 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 LEADER]: Becoming Leader. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:04:19.954031 25648 raft_consensus.cc:504] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:19.970279 25558 consensus_queue.cc:237] T c85f5d1d3f004cf5817bb594fd8e6469 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:19.971246 25648 leader_election.cc:288] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 election: Requested vote from peers eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:19.973023 24616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e725b82985746b39c0cc8f671c95353" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:19.973898 24616 raft_consensus.cc:2454] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 2.
I20260925 03:04:19.974777 25555 raft_consensus.cc:686] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 LEADER]: Becoming Leader. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:04:19.975975 25555 consensus_queue.cc:237] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:19.961041 24541 raft_consensus.cc:2454] T 420163a6cdce412ebc21c25fe3835638 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 3.
I20260925 03:04:19.983237 24617 raft_consensus.cc:2454] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 3.
I20260925 03:04:19.958943 25657 raft_consensus.cc:482] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:19.989476 24622 raft_consensus.cc:3037] T 420163a6cdce412ebc21c25fe3835638 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:19.989676 24467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bb46bce3564841b7674e6a693bd2e6" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:19.992076 24467 raft_consensus.cc:3037] T 11bb46bce3564841b7674e6a693bd2e6 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:19.965296 24618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71f461a6e2ff434993b78a2d077f0abd" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:19.993474 25657 raft_consensus.cc:3037] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:19.986917 25572 raft_consensus.cc:2726] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389) with higher term. Message: Invalid argument: T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c for earlier term 2. Current term is 3.
I20260925 03:04:20.004978 24368 catalog_manager.cc:5495] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43 reported cstate change: term changed from 1 to 2, leader changed from 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1) to 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4). New cstate: current_term: 2 leader_uuid: "1bc1f2e62ab44af995165bd5116a9d43" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260925 03:04:19.991919 24469 raft_consensus.cc:2454] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 3.
I20260925 03:04:20.015062 25199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71f461a6e2ff434993b78a2d077f0abd" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43"
I20260925 03:04:20.015854 25199 raft_consensus.cc:2379] T 71f461a6e2ff434993b78a2d077f0abd P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 3: Already voted for candidate 1bc1f2e62ab44af995165bd5116a9d43 in this term.
I20260925 03:04:20.017642 25640 leader_election.cc:288] T 11bb46bce3564841b7674e6a693bd2e6 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 3 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:20.018779 25516 leader_election.cc:288] T f22ddd7d1c444dc58f45a1fb0b3e02bc P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
W20260925 03:04:20.006788 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 65115.3us
I20260925 03:04:20.007416 24618 raft_consensus.cc:3037] T 71f461a6e2ff434993b78a2d077f0abd P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Advancing to term 3
W20260925 03:04:20.010790 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 61851.6us
I20260925 03:04:20.021216 24514 leader_election.cc:302] T 37a157996e944db2906110d87ecb0c54 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
W20260925 03:04:20.004729 25671 log.cc:924] Time spent T c88cbd39e01f44628a094af516be2833 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.083s user 0.002s sys 0.000s
I20260925 03:04:20.032680 24467 raft_consensus.cc:2454] T 11bb46bce3564841b7674e6a693bd2e6 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 3.
I20260925 03:04:20.038582 24622 raft_consensus.cc:2454] T 420163a6cdce412ebc21c25fe3835638 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 3.
I20260925 03:04:20.039193 25516 raft_consensus.cc:2781] T 37a157996e944db2906110d87ecb0c54 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 03:04:20.037045 25196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15861cd28e354598a3c6bfc4337c34bb" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:20.047346 25196 raft_consensus.cc:2379] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 2: Already voted for candidate 1bc1f2e62ab44af995165bd5116a9d43 in this term.
I20260925 03:04:20.053117 24617 raft_consensus.cc:1226] T 11bb46bce3564841b7674e6a693bd2e6 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260925 03:04:20.053016 25516 raft_consensus.cc:686] T 37a157996e944db2906110d87ecb0c54 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 LEADER]: Becoming Leader. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:04:20.055299 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71f461a6e2ff434993b78a2d077f0abd" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:20.056209 24545 raft_consensus.cc:2379] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1bc1f2e62ab44af995165bd5116a9d43 in current term 3: Already voted for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in this term.
I20260925 03:04:20.056027 25516 consensus_queue.cc:237] T 37a157996e944db2906110d87ecb0c54 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:20.066395 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b030d84c670b48d5bff337ccc0c77626" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:20.067103 24545 raft_consensus.cc:3037] T b030d84c670b48d5bff337ccc0c77626 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Advancing to term 3
W20260925 03:04:20.070533 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 50469.2us
I20260925 03:04:20.071362 24437 leader_election.cc:302] T 06f4a7eaf6534e4986c9fd35911038e4 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:04:20.072332 25584 raft_consensus.cc:2781] T 06f4a7eaf6534e4986c9fd35911038e4 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260925 03:04:20.072799 25584 raft_consensus.cc:482] T 06f4a7eaf6534e4986c9fd35911038e4 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:20.073153 25584 raft_consensus.cc:3037] T 06f4a7eaf6534e4986c9fd35911038e4 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:20.075016 24437 leader_election.cc:302] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:04:20.075968 25642 raft_consensus.cc:2781] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 03:04:20.076481 25642 raft_consensus.cc:686] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 LEADER]: Becoming Leader. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:04:20.076958 24467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c23337a7cb74184a0e57b7095cb6a11" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:20.077659 24469 raft_consensus.cc:1261] T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260925 03:04:20.077726 24467 raft_consensus.cc:2354] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 2. Current term is 3.
I20260925 03:04:20.077838 24464 raft_consensus.cc:1226] T 11bb46bce3564841b7674e6a693bd2e6 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260925 03:04:20.077881 25642 consensus_queue.cc:237] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:20.086665 24514 leader_election.cc:398] T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 election: Vote denied by peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187) with higher term. Message: Invalid argument: T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 2. Current term is 3.
I20260925 03:04:20.086968 24468 raft_consensus.cc:1261] T 4e5332a5d329493991dff994e70ea92c P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260925 03:04:20.087018 24514 leader_election.cc:401] T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260925 03:04:20.093127 24586 leader_election.cc:302] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:04:20.101109 24467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f22ddd7d1c444dc58f45a1fb0b3e02bc" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:20.102039 24467 raft_consensus.cc:2379] T f22ddd7d1c444dc58f45a1fb0b3e02bc P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 2: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:04:20.108136 24544 raft_consensus.cc:1261] T 367277daf8d04778b3fa4ed94d71580d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260925 03:04:20.111578 24617 raft_consensus.cc:1261] T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260925 03:04:20.115633 25555 consensus_queue.cc:1046] T 11bb46bce3564841b7674e6a693bd2e6 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20260925 03:04:20.121608 25657 raft_consensus.cc:504] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:20.122949 24588 leader_election.cc:302] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:20.123780 25657 leader_election.cc:288] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:20.124240 24588 leader_election.cc:302] T 11bb46bce3564841b7674e6a693bd2e6 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:20.124342 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e725b82985746b39c0cc8f671c95353" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:20.125147 24541 raft_consensus.cc:2454] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 2.
I20260925 03:04:20.125602 25636 raft_consensus.cc:2781] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader election won for term 3
I20260925 03:04:20.125864 25645 consensus_queue.cc:1035] T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.010s
I20260925 03:04:20.126686 25557 raft_consensus.cc:2781] T 11bb46bce3564841b7674e6a693bd2e6 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader election won for term 3
I20260925 03:04:20.127192 25557 raft_consensus.cc:686] T 11bb46bce3564841b7674e6a693bd2e6 P eeab93abfdb141a895002ec9f312296c [term 3 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:20.127296 24614 raft_consensus.cc:1261] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260925 03:04:20.128212 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ac285175464fbda67211d9574db415" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:20.128053 25557 consensus_queue.cc:237] T 11bb46bce3564841b7674e6a693bd2e6 P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:20.129519 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37a157996e944db2906110d87ecb0c54" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:20.129699 24615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c23337a7cb74184a0e57b7095cb6a11" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:20.126144 25636 raft_consensus.cc:686] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [term 3 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:20.130251 24618 raft_consensus.cc:2454] T 71f461a6e2ff434993b78a2d077f0abd P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 3.
I20260925 03:04:20.130389 25555 raft_consensus.cc:3032] T 11bb46bce3564841b7674e6a693bd2e6 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 LEADER]: Stepping down as leader of term 2
I20260925 03:04:20.130280 24616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60d45b7d467d4e1ab92367f6d7d52b15" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:20.129022 24621 raft_consensus.cc:2354] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 2. Current term is 3.
I20260925 03:04:20.124490 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71f461a6e2ff434993b78a2d077f0abd" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:20.129067 25560 consensus_queue.cc:1035] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260925 03:04:20.144188 25517 raft_consensus.cc:482] T 8ac466793d094325ace224708b2158ed P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:20.069149 24540 raft_consensus.cc:2454] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 3.
I20260925 03:04:20.144955 25636 consensus_queue.cc:237] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:20.146447 24615 raft_consensus.cc:2379] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 2: Already voted for candidate eeab93abfdb141a895002ec9f312296c in this term.
I20260925 03:04:20.133836 24467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15861cd28e354598a3c6bfc4337c34bb" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:20.108136 25643 raft_consensus.cc:3037] T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:20.112898 25572 raft_consensus.cc:2741] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260925 03:04:20.151448 24586 leader_election.cc:398] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Vote denied by peer a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389) with higher term. Message: Invalid argument: T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c for earlier term 2. Current term is 3.
I20260925 03:04:20.151754 24586 leader_election.cc:401] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260925 03:04:20.143775 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7691c6b475428c82d6181d693d26f3" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:20.144308 24539 raft_consensus.cc:1261] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260925 03:04:20.140631 24436 leader_election.cc:302] T 37a157996e944db2906110d87ecb0c54 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:20.153132 24620 raft_consensus.cc:2379] T 71f461a6e2ff434993b78a2d077f0abd P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 3: Already voted for candidate 1bc1f2e62ab44af995165bd5116a9d43 in this term.
I20260925 03:04:20.139917 25555 raft_consensus.cc:729] T 11bb46bce3564841b7674e6a693bd2e6 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:04:20.148808 25649 raft_consensus.cc:2741] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader election decision vote started in defunct term 2: won
I20260925 03:04:20.144878 25517 raft_consensus.cc:504] T 8ac466793d094325ace224708b2158ed P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:20.162619 25555 consensus_queue.cc:260] T 11bb46bce3564841b7674e6a693bd2e6 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:20.163898 25555 raft_consensus.cc:3037] T 11bb46bce3564841b7674e6a693bd2e6 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:20.164346 25517 leader_election.cc:288] T 8ac466793d094325ace224708b2158ed P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:20.164994 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ac466793d094325ace224708b2158ed" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:20.166515 25162 leader_election.cc:302] T 71f461a6e2ff434993b78a2d077f0abd P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, eeab93abfdb141a895002ec9f312296c; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:20.167281 25517 raft_consensus.cc:2781] T 71f461a6e2ff434993b78a2d077f0abd P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 03:04:20.167721 25517 raft_consensus.cc:686] T 71f461a6e2ff434993b78a2d077f0abd P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 LEADER]: Becoming Leader. State: Replica: 1bc1f2e62ab44af995165bd5116a9d43, State: Running, Role: LEADER
I20260925 03:04:20.168646 25517 consensus_queue.cc:237] T 71f461a6e2ff434993b78a2d077f0abd P 1bc1f2e62ab44af995165bd5116a9d43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:20.155992 24541 raft_consensus.cc:2454] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 2.
I20260925 03:04:20.152197 24613 raft_consensus.cc:3037] T 37a157996e944db2906110d87ecb0c54 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Advancing to term 3
I20260925 03:04:20.144556 24616 raft_consensus.cc:3037] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Advancing to term 3
I20260925 03:04:20.176358 24586 leader_election.cc:302] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:04:20.176709 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ac466793d094325ace224708b2158ed" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:20.149158 25563 raft_consensus.cc:504] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:20.193122 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c23337a7cb74184a0e57b7095cb6a11" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:20.201644 25563 leader_election.cc:288] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 3 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
W20260925 03:04:20.202524 24436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 61956.4us
I20260925 03:04:20.203514 24436 leader_election.cc:302] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:20.204071 25685 raft_consensus.cc:2741] T 37a157996e944db2906110d87ecb0c54 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260925 03:04:20.204349 24436 leader_election.cc:302] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:20.210150 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b5df303cd6a4d59aca5c18bbbff24ed" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:20.210884 24539 raft_consensus.cc:2454] T 0b5df303cd6a4d59aca5c18bbbff24ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 2.
I20260925 03:04:20.211172 24586 leader_election.cc:302] T 37a157996e944db2906110d87ecb0c54 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:20.207907 25640 raft_consensus.cc:2781] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Leader election won for term 2
I20260925 03:04:20.212224 25558 raft_consensus.cc:2781] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260925 03:04:20.212318 25640 raft_consensus.cc:686] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [term 2 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:20.212723 25558 raft_consensus.cc:482] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:20.213068 25558 raft_consensus.cc:3037] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:20.213332 25640 consensus_queue.cc:237] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:20.214991 24586 leader_election.cc:302] T 0b5df303cd6a4d59aca5c18bbbff24ed P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:04:20.215929 25666 raft_consensus.cc:2781] T 0b5df303cd6a4d59aca5c18bbbff24ed P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Leader election won for term 2
I20260925 03:04:20.216437 25666 raft_consensus.cc:686] T 0b5df303cd6a4d59aca5c18bbbff24ed P eeab93abfdb141a895002ec9f312296c [term 2 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:20.216938 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f22ddd7d1c444dc58f45a1fb0b3e02bc" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:20.217561 24466 raft_consensus.cc:3037] T f22ddd7d1c444dc58f45a1fb0b3e02bc P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:20.217381 25666 consensus_queue.cc:237] T 0b5df303cd6a4d59aca5c18bbbff24ed P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:20.218598 24465 raft_consensus.cc:1261] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260925 03:04:20.218724 24586 leader_election.cc:302] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:04:20.219594 25647 raft_consensus.cc:2781] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader election won for term 3
I20260925 03:04:20.221382 24545 raft_consensus.cc:2454] T b030d84c670b48d5bff337ccc0c77626 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 3.
I20260925 03:04:20.224934 24467 raft_consensus.cc:2454] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 2.
I20260925 03:04:20.227880 24540 raft_consensus.cc:1261] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260925 03:04:20.228569 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bb46bce3564841b7674e6a693bd2e6" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:20.220062 25647 raft_consensus.cc:686] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [term 3 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:20.232880 25535 consensus_queue.cc:1035] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.004s
I20260925 03:04:20.230681 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f22ddd7d1c444dc58f45a1fb0b3e02bc" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:20.233286 25647 consensus_queue.cc:237] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:20.236200 24540 raft_consensus.cc:3037] T f22ddd7d1c444dc58f45a1fb0b3e02bc P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:20.210908 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5f9b503b0ab43999d582015a367441e" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:20.231374 24548 raft_consensus.cc:1261] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260925 03:04:20.239986 24514 leader_election.cc:302] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:04:20.239445 24541 raft_consensus.cc:2454] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 2.
I20260925 03:04:20.241149 25162 leader_election.cc:302] T 8ac466793d094325ace224708b2158ed P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:20.244474 24371 catalog_manager.cc:5495] T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c reported cstate change: term changed from 1 to 2, leader changed from a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2) to eeab93abfdb141a895002ec9f312296c (127.22.193.3). New cstate: current_term: 2 leader_uuid: "eeab93abfdb141a895002ec9f312296c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: HEALTHY } } }
W20260925 03:04:20.305102 24436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 70410.7us
I20260925 03:04:20.305905 24371 catalog_manager.cc:5495] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c reported cstate change: term changed from 1 to 2, leader changed from a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2) to eeab93abfdb141a895002ec9f312296c (127.22.193.3). New cstate: current_term: 2 leader_uuid: "eeab93abfdb141a895002ec9f312296c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: HEALTHY } } }
I20260925 03:04:20.243861 25641 raft_consensus.cc:2781] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 03:04:20.275341 25516 raft_consensus.cc:2781] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 03:04:20.329828 24371 catalog_manager.cc:5495] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c reported cstate change: term changed from 1 to 2, leader changed from a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2) to eeab93abfdb141a895002ec9f312296c (127.22.193.3). New cstate: current_term: 2 leader_uuid: "eeab93abfdb141a895002ec9f312296c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: HEALTHY } } }
I20260925 03:04:20.330947 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c23337a7cb74184a0e57b7095cb6a11" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:20.250077 25666 consensus_queue.cc:1035] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.070s
I20260925 03:04:20.338220 25517 raft_consensus.cc:2726] T 8ac466793d094325ace224708b2158ed P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260925 03:04:20.315666 25643 raft_consensus.cc:2726] T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187) with higher term. Message: Invalid argument: T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 2. Current term is 3.
I20260925 03:04:20.395615 24618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7691c6b475428c82d6181d693d26f3" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:20.396100 24619 raft_consensus.cc:3037] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Advancing to term 3
W20260925 03:04:20.338374 24355 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.GetConsensusState -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 56074.8us
I20260925 03:04:20.402163 24367 catalog_manager.cc:5495] T c88cbd39e01f44628a094af516be2833 P a04a37fc4aa7460cbfdbeeb4e451e1f5 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: UNKNOWN } } }
W20260925 03:04:20.403854 25693 log.cc:924] Time spent T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5: Append to log took a long time: real 0.072s user 0.000s sys 0.000s
I20260925 03:04:20.331511 24614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b5df303cd6a4d59aca5c18bbbff24ed" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:20.431948 24620 raft_consensus.cc:1261] T 367277daf8d04778b3fa4ed94d71580d P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260925 03:04:20.335106 25516 raft_consensus.cc:686] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 LEADER]: Becoming Leader. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:04:20.304962 25636 consensus_queue.cc:1035] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.021s
I20260925 03:04:20.316740 25622 raft_consensus.cc:482] T a4abe0ea757e4886be4945c5001e529a P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:20.397639 24436 leader_election.cc:302] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5
W20260925 03:04:20.444161 25162 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 212494us
I20260925 03:04:20.450778 25640 raft_consensus.cc:482] T 3d89396576884f7c8901dba54f783947 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:20.455914 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5f9b503b0ab43999d582015a367441e" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:20.456562 24615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f22ddd7d1c444dc58f45a1fb0b3e02bc" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:20.475019 25640 raft_consensus.cc:504] T 3d89396576884f7c8901dba54f783947 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:20.477110 24615 raft_consensus.cc:2354] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 2. Current term is 3.
I20260925 03:04:20.478463 24615 raft_consensus.cc:1261] T 74443a0a1dc34d88a2f42e3b722db405 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260925 03:04:20.459064 24622 raft_consensus.cc:1261] T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260925 03:04:20.459367 24367 catalog_manager.cc:5495] T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: UNKNOWN } } }
I20260925 03:04:20.481503 25640 leader_election.cc:288] T 3d89396576884f7c8901dba54f783947 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:20.482003 24367 catalog_manager.cc:5495] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: UNKNOWN } } }
I20260925 03:04:20.483692 24367 catalog_manager.cc:5495] T 442d4a29b1d743599f4b6b4871e26668 P a04a37fc4aa7460cbfdbeeb4e451e1f5 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: UNKNOWN } } }
I20260925 03:04:20.460446 24547 raft_consensus.cc:2396] T 11bb46bce3564841b7674e6a693bd2e6 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260925 03:04:20.485299 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d89396576884f7c8901dba54f783947" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:20.490093 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d89396576884f7c8901dba54f783947" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:20.495175 24588 leader_election.cc:302] T 3d89396576884f7c8901dba54f783947 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:20.488492 24367 catalog_manager.cc:5495] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: UNKNOWN } } }
I20260925 03:04:20.503484 25199 raft_consensus.cc:1261] T 7e49248643df4131a1c42e74244e05c2 P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 LEARNER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260925 03:04:20.508028 25703 raft_consensus.cc:2726] T 3d89396576884f7c8901dba54f783947 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 03:04:20.508512 25658 raft_consensus.cc:2781] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260925 03:04:20.508985 25658 raft_consensus.cc:482] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:20.509311 25658 raft_consensus.cc:3037] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Advancing to term 3
W20260925 03:04:20.516463 25689 log.cc:924] Time spent T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5: Append to log took a long time: real 0.200s user 0.001s sys 0.000s
I20260925 03:04:20.517804 25165 leader_election.cc:302] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:20.518711 25517 raft_consensus.cc:2781] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260925 03:04:20.519102 25517 raft_consensus.cc:482] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:20.519423 25517 raft_consensus.cc:3037] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:20.520033 25557 raft_consensus.cc:482] T 43366a0a2f9d4344bf4e37ffd290ef8a P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:20.520614 25557 raft_consensus.cc:504] T 43366a0a2f9d4344bf4e37ffd290ef8a P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:20.524361 24616 raft_consensus.cc:2454] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 3.
W20260925 03:04:20.525792 25695 log.cc:924] Time spent T 0b5df303cd6a4d59aca5c18bbbff24ed P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.094s user 0.000s sys 0.000s
I20260925 03:04:20.535753 24544 raft_consensus.cc:2454] T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 3.
I20260925 03:04:20.560884 24468 raft_consensus.cc:1261] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20260925 03:04:20.563233 24436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.4:43979, user_credentials={real_user=slave}} blocked reactor thread for 165720us
W20260925 03:04:20.524365 25690 log.cc:924] Time spent T 37a157996e944db2906110d87ecb0c54 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.122s user 0.001s sys 0.000s
I20260925 03:04:20.529243 25572 raft_consensus.cc:2726] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389) with higher term. Message: Invalid argument: T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c for earlier term 2. Current term is 3.
I20260925 03:04:20.571483 25196 raft_consensus.cc:3037] T 367277daf8d04778b3fa4ed94d71580d P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 LEARNER]: Advancing to term 3
I20260925 03:04:20.577086 25555 consensus_queue.cc:1035] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260925 03:04:20.578842 24511 leader_election.cc:398] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 election: Vote denied by peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105) with higher term. Message: Invalid argument: T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 2. Current term is 3.
I20260925 03:04:20.470327 25584 raft_consensus.cc:504] T 06f4a7eaf6534e4986c9fd35911038e4 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:20.579764 24511 leader_election.cc:302] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 1bc1f2e62ab44af995165bd5116a9d43, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:20.529048 25622 raft_consensus.cc:504] T a4abe0ea757e4886be4945c5001e529a P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:20.585271 25710 consensus_queue.cc:1035] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260925 03:04:20.586972 25622 leader_election.cc:288] T a4abe0ea757e4886be4945c5001e529a P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:20.587572 25516 consensus_queue.cc:237] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
W20260925 03:04:20.587040 25691 log.cc:924] Time spent T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.181s user 0.000s sys 0.001s
I20260925 03:04:20.600447 24511 leader_election.cc:398] T f22ddd7d1c444dc58f45a1fb0b3e02bc P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 election: Vote denied by peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105) with higher term. Message: Invalid argument: T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 2. Current term is 3.
I20260925 03:04:20.600898 24511 leader_election.cc:401] T f22ddd7d1c444dc58f45a1fb0b3e02bc P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
W20260925 03:04:20.602689 25694 log.cc:924] Time spent T 4e5332a5d329493991dff994e70ea92c P 869b0a8575b54d08ae2817fad5dd42d5: Append to log took a long time: real 0.051s user 0.001s sys 0.000s
I20260925 03:04:20.603631 24472 raft_consensus.cc:1226] T 06f4a7eaf6534e4986c9fd35911038e4 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 2. Current term is 3. Ops: []
I20260925 03:04:20.608153 25697 raft_consensus.cc:482] T b1e85248d3b04b979dd080caeeb02fdb P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:20.608820 25697 raft_consensus.cc:504] T b1e85248d3b04b979dd080caeeb02fdb P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:20.611334 25697 leader_election.cc:288] T b1e85248d3b04b979dd080caeeb02fdb P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:20.605554 25642 consensus_queue.cc:1035] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.007s
I20260925 03:04:20.618033 25584 leader_election.cc:288] T 06f4a7eaf6534e4986c9fd35911038e4 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:20.622614 25658 raft_consensus.cc:504] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:20.630519 25678 consensus_queue.cc:1035] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.047s
I20260925 03:04:20.596297 25702 consensus_queue.cc:1035] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.054s
W20260925 03:04:20.643862 24511 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 43586.3us
I20260925 03:04:20.650092 25649 consensus_queue.cc:1035] T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260925 03:04:20.651845 25649 raft_consensus.cc:482] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:20.602738 25643 raft_consensus.cc:2726] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260925 03:04:20.652366 25649 raft_consensus.cc:504] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:20.655457 25557 leader_election.cc:288] T 43366a0a2f9d4344bf4e37ffd290ef8a P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
W20260925 03:04:20.656941 25692 log.cc:924] Time spent T abd5b8dc1b7e49e3aa9af900d73ff0e7 P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.220s user 0.000s sys 0.002s
I20260925 03:04:20.658398 25658 leader_election.cc:288] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:20.656831 25649 leader_election.cc:288] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:20.659227 24469 raft_consensus.cc:1261] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260925 03:04:20.662377 25572 consensus_queue.cc:1035] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260925 03:04:20.664965 24469 raft_consensus.cc:1261] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260925 03:04:20.666078 24466 raft_consensus.cc:2454] T f22ddd7d1c444dc58f45a1fb0b3e02bc P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 3.
I20260925 03:04:20.668792 24617 raft_consensus.cc:1261] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260925 03:04:20.668771 25710 consensus_queue.cc:1046] T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260925 03:04:20.670184 24616 raft_consensus.cc:1261] T 442d4a29b1d743599f4b6b4871e26668 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260925 03:04:20.671861 24621 raft_consensus.cc:1261] T c88cbd39e01f44628a094af516be2833 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260925 03:04:20.672950 25558 raft_consensus.cc:504] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:20.674499 24615 raft_consensus.cc:1261] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260925 03:04:20.677058 25563 consensus_queue.cc:1035] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260925 03:04:20.679204 24614 raft_consensus.cc:1261] T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260925 03:04:20.652462 24541 raft_consensus.cc:1261] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260925 03:04:20.683542 24619 raft_consensus.cc:2454] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 3.
I20260925 03:04:20.688560 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43366a0a2f9d4344bf4e37ffd290ef8a" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:20.694481 24588 leader_election.cc:302] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:20.697316 24545 raft_consensus.cc:1261] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260925 03:04:20.699928 24615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:20.700572 24615 raft_consensus.cc:2454] T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 3.
I20260925 03:04:20.701715 25198 raft_consensus.cc:3032] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 LEADER]: Stepping down as leader of term 2
I20260925 03:04:20.702059 25198 raft_consensus.cc:729] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1bc1f2e62ab44af995165bd5116a9d43, State: Running, Role: LEADER
I20260925 03:04:20.644245 24540 raft_consensus.cc:2454] T f22ddd7d1c444dc58f45a1fb0b3e02bc P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 3.
I20260925 03:04:20.704455 25198 consensus_queue.cc:260] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:20.768332 25198 raft_consensus.cc:3037] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 FOLLOWER]: Advancing to term 3
W20260925 03:04:20.771262 24511 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 63062.3us
W20260925 03:04:20.775936 24436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 57551.3us
I20260925 03:04:20.783486 25198 raft_consensus.cc:1261] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 5. (term mismatch)
I20260925 03:04:20.784145 25198 pending_rounds.cc:76] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43: Aborting all ops after (but not including) 4
I20260925 03:04:20.697697 25641 raft_consensus.cc:686] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 LEADER]: Becoming Leader. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:04:20.707475 25560 raft_consensus.cc:2726] T f22ddd7d1c444dc58f45a1fb0b3e02bc P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105) with higher term. Message: Invalid argument: T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 2. Current term is 3.
I20260925 03:04:20.787950 25198 pending_rounds.cc:97] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43: Aborting uncommitted NO_OP operation due to leader change: 2.5
I20260925 03:04:20.817538 25198 raft_consensus.cc:2866] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260925 03:04:20.776520 25710 raft_consensus.cc:2781] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader election won for term 3
I20260925 03:04:20.834000 24467 raft_consensus.cc:1261] T 11bb46bce3564841b7674e6a693bd2e6 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260925 03:04:20.855598 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06f4a7eaf6534e4986c9fd35911038e4" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:20.798679 25649 consensus_queue.cc:1035] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.061s
I20260925 03:04:20.770927 25647 consensus_queue.cc:1035] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.060s
I20260925 03:04:20.827961 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4abe0ea757e4886be4945c5001e529a" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:20.809893 25678 consensus_queue.cc:1035] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.049s
I20260925 03:04:20.815948 25664 consensus_queue.cc:1035] T c88cbd39e01f44628a094af516be2833 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.035s
I20260925 03:04:20.871167 25199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15861cd28e354598a3c6bfc4337c34bb" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43"
I20260925 03:04:20.817498 25643 consensus_queue.cc:1035] T 442d4a29b1d743599f4b6b4871e26668 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.035s
I20260925 03:04:20.878298 24468 raft_consensus.cc:1261] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260925 03:04:20.882438 25539 consensus_queue.cc:1035] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.186s
I20260925 03:04:20.883935 25705 raft_consensus.cc:482] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:20.885106 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:20.862208 25710 raft_consensus.cc:686] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c [term 3 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:20.896499 24473 raft_consensus.cc:1261] T 442d4a29b1d743599f4b6b4871e26668 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260925 03:04:20.846472 25636 consensus_queue.cc:1035] T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.029s
I20260925 03:04:20.899711 25198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15861cd28e354598a3c6bfc4337c34bb" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43"
I20260925 03:04:20.901943 24615 raft_consensus.cc:1261] T 420163a6cdce412ebc21c25fe3835638 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260925 03:04:20.902398 24471 raft_consensus.cc:1261] T c88cbd39e01f44628a094af516be2833 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260925 03:04:20.819068 25706 consensus_queue.cc:1035] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.097s
I20260925 03:04:20.907325 24468 raft_consensus.cc:1261] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260925 03:04:20.884500 25705 raft_consensus.cc:504] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:20.910799 24437 leader_election.cc:302] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:04:20.910427 25702 consensus_queue.cc:1035] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.003s
I20260925 03:04:20.906924 24467 raft_consensus.cc:1261] T b627ecb4a5384ef392aa695e06599c39 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260925 03:04:20.899170 25555 raft_consensus.cc:482] T 3d89396576884f7c8901dba54f783947 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:20.913259 25710 consensus_queue.cc:237] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:20.914023 24371 catalog_manager.cc:5495] T 71f461a6e2ff434993b78a2d077f0abd P 1bc1f2e62ab44af995165bd5116a9d43 reported cstate change: term changed from 1 to 3, leader changed from a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2) to 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4). New cstate: current_term: 3 leader_uuid: "1bc1f2e62ab44af995165bd5116a9d43" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260925 03:04:20.917718 25698 raft_consensus.cc:2781] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260925 03:04:20.918201 25698 raft_consensus.cc:482] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:20.918532 25698 raft_consensus.cc:3037] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Advancing to term 3
W20260925 03:04:20.919296 25585 log.cc:924] Time spent T 00000000000000000000000000000000 P 2d92cc0732c4490eb7d98d353826cce1: Append to log took a long time: real 0.160s user 0.002s sys 0.000s
I20260925 03:04:20.913834 24472 raft_consensus.cc:1261] T 37a157996e944db2906110d87ecb0c54 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260925 03:04:20.896762 25572 raft_consensus.cc:3032] T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c [term 2 LEADER]: Stepping down as leader of term 2
I20260925 03:04:20.923377 25705 leader_election.cc:288] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:20.917364 25555 raft_consensus.cc:504] T 3d89396576884f7c8901dba54f783947 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:20.925246 24616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4abe0ea757e4886be4945c5001e529a" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:20.925402 24437 leader_election.cc:302] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:04:20.925734 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15861cd28e354598a3c6bfc4337c34bb" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:20.911492 25558 leader_election.cc:288] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:20.932395 25698 raft_consensus.cc:504] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:20.935806 25195 raft_consensus.cc:1261] T 29637f5a4a504ce6ba29033ccd0cd716 P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 LEARNER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260925 03:04:20.875885 25641 consensus_queue.cc:237] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:20.939138 24437 leader_election.cc:302] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:04:20.900281 25539 consensus_queue.cc:1035] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.039s
I20260925 03:04:20.939632 25700 raft_consensus.cc:2781] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260925 03:04:20.928649 25572 raft_consensus.cc:729] T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:20.929006 24615 raft_consensus.cc:1226] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [term 3 LEADER]: Rejecting Update request from peer 1bc1f2e62ab44af995165bd5116a9d43 for earlier term 2. Current term is 3. Ops: []
I20260925 03:04:20.959992 24437 leader_election.cc:302] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:04:20.961565 25198 raft_consensus.cc:1261] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260925 03:04:20.953473 25698 leader_election.cc:288] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:20.966815 24542 raft_consensus.cc:1261] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Refusing update from remote peer 1bc1f2e62ab44af995165bd5116a9d43: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260925 03:04:20.967967 25572 consensus_queue.cc:260] T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:20.973418 24620 raft_consensus.cc:1261] T c85f5d1d3f004cf5817bb594fd8e6469 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260925 03:04:20.951489 25558 raft_consensus.cc:2781] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260925 03:04:20.950788 25700 raft_consensus.cc:482] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:20.968626 25196 raft_consensus.cc:1261] T 367277daf8d04778b3fa4ed94d71580d P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 LEARNER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260925 03:04:20.977814 25658 raft_consensus.cc:2781] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 03:04:20.946807 25517 raft_consensus.cc:504] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:20.981881 25700 raft_consensus.cc:3037] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:20.969853 25192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:20.985527 25622 consensus_queue.cc:1035] T 71f461a6e2ff434993b78a2d077f0abd P 1bc1f2e62ab44af995165bd5116a9d43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.009s
I20260925 03:04:20.969292 25193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e54ca1757ee4e42956a9cebbeca4f62" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:20.995935 25712 raft_consensus.cc:482] T b627ecb4a5384ef392aa695e06599c39 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:20.996474 25712 raft_consensus.cc:504] T b627ecb4a5384ef392aa695e06599c39 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:20.982975 25558 raft_consensus.cc:482] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:21.000164 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1e85248d3b04b979dd080caeeb02fdb" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:20.989286 25572 raft_consensus.cc:3037] T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:21.002403 25647 consensus_queue.cc:1035] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.002s
I20260925 03:04:21.003820 25162 leader_election.cc:302] T a4abe0ea757e4886be4945c5001e529a P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:20.961143 25666 consensus_queue.cc:1035] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.030s
I20260925 03:04:20.982029 25678 consensus_queue.cc:1035] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.022s
I20260925 03:04:21.008733 25199 raft_consensus.cc:2379] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 in current term 3: Already voted for candidate 1bc1f2e62ab44af995165bd5116a9d43 in this term.
W20260925 03:04:21.002004 24613 consensus_meta.cc:287] T 37a157996e944db2906110d87ecb0c54 P eeab93abfdb141a895002ec9f312296c: Time spent flushing consensus metadata: real 0.801s user 0.010s sys 0.002s
I20260925 03:04:20.987212 25706 consensus_queue.cc:1035] T c88cbd39e01f44628a094af516be2833 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.020s
I20260925 03:04:20.986249 25664 consensus_queue.cc:1035] T 442d4a29b1d743599f4b6b4871e26668 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.016s
I20260925 03:04:20.989439 25705 consensus_queue.cc:1035] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.020s
I20260925 03:04:21.006233 25555 leader_election.cc:288] T 3d89396576884f7c8901dba54f783947 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:21.001952 25558 raft_consensus.cc:3037] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:21.012898 25622 raft_consensus.cc:2726] T a4abe0ea757e4886be4945c5001e529a P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 03:04:20.988996 25643 consensus_queue.cc:1035] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.019s
I20260925 03:04:20.999115 25712 leader_election.cc:288] T b627ecb4a5384ef392aa695e06599c39 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:21.006649 25658 raft_consensus.cc:686] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 LEADER]: Becoming Leader. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:04:21.013864 24613 raft_consensus.cc:2454] T 37a157996e944db2906110d87ecb0c54 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 3.
I20260925 03:04:21.012123 25649 consensus_queue.cc:1035] T 37a157996e944db2906110d87ecb0c54 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.030s
I20260925 03:04:21.066916 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e725b82985746b39c0cc8f671c95353" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:21.067595 24542 raft_consensus.cc:3037] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:21.075028 24511 leader_election.cc:302] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:04:21.066198 25639 raft_consensus.cc:2726] T 37a157996e944db2906110d87ecb0c54 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 03:04:21.072034 25709 raft_consensus.cc:482] T 367277daf8d04778b3fa4ed94d71580d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:21.094918 24473 raft_consensus.cc:1261] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
W20260925 03:04:21.101812 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 112497us
I20260925 03:04:21.092293 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7691c6b475428c82d6181d693d26f3" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:21.110116 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e54ca1757ee4e42956a9cebbeca4f62" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:21.115264 24471 raft_consensus.cc:1226] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 2. Current term is 3. Ops: []
I20260925 03:04:21.157579 24472 raft_consensus.cc:1226] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 2. Current term is 3. Ops: []
I20260925 03:04:21.074976 25658 consensus_queue.cc:237] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:21.106812 24618 raft_consensus.cc:1261] T 37a157996e944db2906110d87ecb0c54 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260925 03:04:21.168923 24544 raft_consensus.cc:1261] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260925 03:04:21.171468 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d89396576884f7c8901dba54f783947" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:21.157373 25517 leader_election.cc:288] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 3 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
W20260925 03:04:21.200757 25717 log.cc:924] Time spent T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.097s user 0.000s sys 0.000s
I20260925 03:04:21.111449 25703 consensus_queue.cc:1035] T 11bb46bce3564841b7674e6a693bd2e6 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.090s
I20260925 03:04:21.169997 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d89396576884f7c8901dba54f783947" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:21.157605 25563 consensus_queue.cc:1035] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.052s
I20260925 03:04:21.202615 25729 consensus_queue.cc:1035] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.004s
I20260925 03:04:21.100440 25709 raft_consensus.cc:504] T 367277daf8d04778b3fa4ed94d71580d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } }
W20260925 03:04:21.237316 24511 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.4:43979, user_credentials={real_user=slave}} blocked reactor thread for 161134us
I20260925 03:04:21.252668 25714 raft_consensus.cc:482] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:21.253326 25714 raft_consensus.cc:504] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:21.259445 25194 raft_consensus.cc:1226] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 2. Current term is 3. Ops: []
I20260925 03:04:21.286100 25714 leader_election.cc:288] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
W20260925 03:04:21.288352 25726 log.cc:924] Time spent T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.069s user 0.000s sys 0.000s
I20260925 03:04:21.267542 25649 raft_consensus.cc:482] T 8e5819c15eac4544a57e5ee6c9d4f117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:21.277542 24539 raft_consensus.cc:1261] T 0b5df303cd6a4d59aca5c18bbbff24ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260925 03:04:21.303753 25727 raft_consensus.cc:482] T 550118bc1a0641ce9803bdbf43f2069f P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:21.307559 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e54ca1757ee4e42956a9cebbeca4f62" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:21.308557 25649 raft_consensus.cc:504] T 8e5819c15eac4544a57e5ee6c9d4f117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:21.368016 24542 raft_consensus.cc:2454] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 3.
I20260925 03:04:21.369835 24543 raft_consensus.cc:1226] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 2. Current term is 3. Ops: []
I20260925 03:04:21.363919 24621 raft_consensus.cc:2454] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 2.
I20260925 03:04:21.390141 25162 leader_election.cc:302] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:21.391535 25714 raft_consensus.cc:2781] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260925 03:04:21.392248 25714 raft_consensus.cc:482] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:21.392767 25714 raft_consensus.cc:3037] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:21.379803 25558 raft_consensus.cc:504] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:21.365986 25697 raft_consensus.cc:482] T e6b6ec010caf4689b8c90449764f5cf9 P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:21.399405 25697 raft_consensus.cc:504] T e6b6ec010caf4689b8c90449764f5cf9 P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:21.402101 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06f4a7eaf6534e4986c9fd35911038e4" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:21.410192 25558 leader_election.cc:288] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:21.366184 25727 raft_consensus.cc:504] T 550118bc1a0641ce9803bdbf43f2069f P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:21.414708 25727 leader_election.cc:288] T 550118bc1a0641ce9803bdbf43f2069f P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:21.422079 24615 raft_consensus.cc:1261] T 71f461a6e2ff434993b78a2d077f0abd P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Refusing update from remote peer 1bc1f2e62ab44af995165bd5116a9d43: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260925 03:04:21.423679 24465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43366a0a2f9d4344bf4e37ffd290ef8a" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:21.425393 25517 consensus_queue.cc:1035] T 71f461a6e2ff434993b78a2d077f0abd P 1bc1f2e62ab44af995165bd5116a9d43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260925 03:04:21.400266 25709 leader_election.cc:288] T 367277daf8d04778b3fa4ed94d71580d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
W20260925 03:04:21.434461 25724 log.cc:924] Time spent T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.522s user 0.001s sys 0.003s
I20260925 03:04:21.434799 25702 raft_consensus.cc:482] T 293ef69caf514ac98510a68495f54eda P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1bc1f2e62ab44af995165bd5116a9d43)
I20260925 03:04:21.407171 25572 raft_consensus.cc:482] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:21.439527 25572 raft_consensus.cc:504] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:21.441908 25697 leader_election.cc:288] T e6b6ec010caf4689b8c90449764f5cf9 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:21.441701 25572 leader_election.cc:288] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:21.446280 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b627ecb4a5384ef392aa695e06599c39" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:21.448132 25649 leader_election.cc:288] T 8e5819c15eac4544a57e5ee6c9d4f117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:21.435827 25702 raft_consensus.cc:504] T 293ef69caf514ac98510a68495f54eda P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:21.461697 25700 raft_consensus.cc:504] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:21.509491 25702 leader_election.cc:288] T 293ef69caf514ac98510a68495f54eda P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:21.537268 25640 raft_consensus.cc:482] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:21.575425 25712 raft_consensus.cc:482] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:21.583639 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15861cd28e354598a3c6bfc4337c34bb" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:21.584192 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1e85248d3b04b979dd080caeeb02fdb" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:21.584537 24468 raft_consensus.cc:2379] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1bc1f2e62ab44af995165bd5116a9d43 in current term 3: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:04:21.588003 24467 raft_consensus.cc:1226] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 2. Current term is 3. Ops: []
I20260925 03:04:21.621955 25639 raft_consensus.cc:482] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:21.622627 25639 raft_consensus.cc:504] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:21.622839 25714 raft_consensus.cc:504] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:21.624766 24367 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39374 (request call id 203) took 1477 ms. Trace:
I20260925 03:04:21.581353 25640 raft_consensus.cc:504] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:21.596906 25712 raft_consensus.cc:504] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } }
I20260925 03:04:21.626570 25700 leader_election.cc:288] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:21.631804 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e54ca1757ee4e42956a9cebbeca4f62" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:21.632303 25712 leader_election.cc:288] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:21.624593 25639 leader_election.cc:288] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:21.633221 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e54ca1757ee4e42956a9cebbeca4f62" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:21.633996 25640 leader_election.cc:288] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:21.634080 25194 raft_consensus.cc:1226] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 2. Current term is 3. Ops: []
I20260925 03:04:21.637714 24465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6b6ec010caf4689b8c90449764f5cf9" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:21.638299 24368 catalog_manager.cc:5495] T c85f5d1d3f004cf5817bb594fd8e6469 P 869b0a8575b54d08ae2817fad5dd42d5 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "869b0a8575b54d08ae2817fad5dd42d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: UNKNOWN } } }
I20260925 03:04:21.639838 24368 catalog_manager.cc:5495] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "869b0a8575b54d08ae2817fad5dd42d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: UNKNOWN } } }
I20260925 03:04:21.662434 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e725b82985746b39c0cc8f671c95353" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:21.663156 25194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ac466793d094325ace224708b2158ed" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:21.663740 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7691c6b475428c82d6181d693d26f3" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:21.668777 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b627ecb4a5384ef392aa695e06599c39" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:21.690238 24436 leader_election.cc:302] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: 1bc1f2e62ab44af995165bd5116a9d43, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:21.692582 25642 raft_consensus.cc:2726] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260925 03:04:21.693135 25198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "293ef69caf514ac98510a68495f54eda" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:21.675302 25714 leader_election.cc:288] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 3 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:21.694679 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5f9b503b0ab43999d582015a367441e" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:21.695796 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b5df303cd6a4d59aca5c18bbbff24ed" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:21.720885 25165 leader_election.cc:302] T b1e85248d3b04b979dd080caeeb02fdb P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:21.723251 25517 raft_consensus.cc:2726] T b1e85248d3b04b979dd080caeeb02fdb P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 03:04:21.725821 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e54ca1757ee4e42956a9cebbeca4f62" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:21.726570 24621 raft_consensus.cc:3037] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:21.719249 25642 raft_consensus.cc:482] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:21.730068 25642 raft_consensus.cc:504] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:21.732184 25642 leader_election.cc:288] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:21.719533 25702 consensus_queue.cc:1035] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.014s
I20260925 03:04:21.625154 24367 rpcz_store.cc:276] 0925 03:04:20.121925 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:20.124892 (+ 2967us) service_pool.cc:227] Handling call
0925 03:04:20.177574 (+ 52682us) spinlock_profiling.cc:243] Waited 40.5 ms on lock 0x7b54001509d4. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455af0 00007f43764d8f2a 00007f437c42ad6c 00007f437c524c3e 00007f437a836fb4 00007f437a836f42 00007f437a836e71 00007f437a836dec 00007f437a8360a2 00007f43748fa94c 00007f43748f9d86
0925 03:04:21.599707 (+1422133us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0925 03:04:20.650299 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0925 03:04:20.650822 (+ 523us) write_op.cc:432] Acquiring schema lock in shared mode
0925 03:04:20.650860 (+ 38us) write_op.cc:435] Acquired schema lock
0925 03:04:20.650874 (+ 14us) tablet.cc:660] Decoding operations
0925 03:04:20.651559 (+ 685us) write_op.cc:620] Acquiring the partition lock for write op
0925 03:04:20.651614 (+ 55us) write_op.cc:641] Partition lock acquired for write op
0925 03:04:20.651641 (+ 27us) tablet.cc:683] Acquiring locks for 5 operations
0925 03:04:20.652210 (+ 569us) tablet.cc:699] Row locks acquired
0925 03:04:20.652228 (+ 18us) write_op.cc:260] PREPARE: finished
0925 03:04:20.652367 (+ 139us) write_op.cc:270] Start()
0925 03:04:20.652477 (+ 110us) write_op.cc:276] Timestamp: P: 1790305460652326 usec, L: 0
0925 03:04:20.652497 (+ 20us) op_driver.cc:331] REPLICATION: starting
0925 03:04:20.704337 (+ 51840us) log.cc:842] Serialized 2078 byte log entry
0925 03:04:20.922294 (+217957us) op_driver.cc:444] REPLICATION: finished
0925 03:04:20.922715 (+ 421us) write_op.cc:301] APPLY: starting
0925 03:04:20.922784 (+ 69us) tablet.cc:1365] starting BulkCheckPresence
0925 03:04:20.922948 (+ 164us) tablet.cc:1368] finished BulkCheckPresence
0925 03:04:20.923047 (+ 99us) tablet.cc:1370] starting ApplyRowOperation cycle
0925 03:04:20.923638 (+ 591us) tablet.cc:1381] finished ApplyRowOperation cycle
0925 03:04:20.923804 (+ 166us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=5
0925 03:04:20.923842 (+ 38us) write_op.cc:312] APPLY: finished
0925 03:04:20.924035 (+ 193us) log.cc:842] Serialized 46 byte log entry
0925 03:04:20.924186 (+ 151us) write_op.cc:489] Releasing partition, row and schema locks
0925 03:04:20.924277 (+ 91us) write_op.cc:454] Released schema lock
0925 03:04:20.924314 (+ 37us) write_op.cc:341] FINISH: Updating metrics
0925 03:04:20.924556 (+ 242us) write_op.cc:489] Releasing partition, row and schema locks
0925 03:04:20.924581 (+ 25us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":5,"spinlock_wait_cycles":93065600,"child_traces":[["op",{"apply.queue_time_us":229,"apply.run_cpu_time_us":2165,"apply.run_wall_time_us":2165,"num_ops":5,"prepare.queue_time_us":30046,"prepare.run_cpu_time_us":3519,"prepare.run_wall_time_us":59743,"replication_time_us":269708}]]}
I20260925 03:04:21.743526 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:21.752591 24621 raft_consensus.cc:2454] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 3.
I20260925 03:04:21.754406 25162 leader_election.cc:302] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:21.755393 25697 raft_consensus.cc:2781] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 03:04:21.755891 25697 raft_consensus.cc:686] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 LEADER]: Becoming Leader. State: Replica: 1bc1f2e62ab44af995165bd5116a9d43, State: Running, Role: LEADER
I20260925 03:04:21.756816 25697 consensus_queue.cc:237] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
W20260925 03:04:21.773859 24436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.4:43979, user_credentials={real_user=slave}} blocked reactor thread for 45076.5us
W20260925 03:04:21.774837 24617 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.603s user 0.025s sys 0.050s
W20260925 03:04:21.778400 24617 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.607s user 0.025s sys 0.050s
W20260925 03:04:21.773681 25641 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.803s user 0.012s sys 0.019s
W20260925 03:04:21.780256 25641 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.809s user 0.012s sys 0.019s
W20260925 03:04:21.781144 24588 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.578s user 0.006s sys 0.016s
W20260925 03:04:21.781558 24588 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.605s user 0.006s sys 0.016s
W20260925 03:04:21.781661 24613 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.573s user 0.011s sys 0.047s
W20260925 03:04:21.782670 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 614161us
I20260925 03:04:21.783730 25637 consensus_queue.cc:1046] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260925 03:04:21.782083 24613 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.576s user 0.011s sys 0.047s
I20260925 03:04:21.785144 25637 raft_consensus.cc:3032] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [term 2 LEADER]: Stepping down as leader of term 2
I20260925 03:04:21.785563 25637 raft_consensus.cc:729] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:21.785876 24588 leader_election.cc:302] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:04:21.786338 25637 consensus_queue.cc:260] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:21.787487 25637 raft_consensus.cc:3037] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:21.788072 25738 consensus_queue.cc:1035] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260925 03:04:21.790807 24622 raft_consensus.cc:1226] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 2. Current term is 3. Ops: []
W20260925 03:04:21.790813 24514 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.668s user 0.002s sys 0.048s
W20260925 03:04:21.791486 24514 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.669s user 0.002s sys 0.048s
W20260925 03:04:21.801792 24547 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.593s user 0.008s sys 0.008s
W20260925 03:04:21.802242 24547 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.594s user 0.008s sys 0.008s
I20260925 03:04:21.802901 25555 raft_consensus.cc:2726] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260925 03:04:21.811223 25754 raft_consensus.cc:2726] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260925 03:04:21.773036 24469 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.650s user 0.000s sys 0.024s
I20260925 03:04:21.767292 25685 consensus_queue.cc:1046] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.009s
W20260925 03:04:21.887302 25199 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.670s user 0.010s sys 0.025s
W20260925 03:04:21.887162 24472 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.645s user 0.009s sys 0.027s
W20260925 03:04:21.818172 25196 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.583s user 0.003s sys 0.028s
W20260925 03:04:21.943212 25199 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.725s user 0.010s sys 0.033s
W20260925 03:04:21.943570 24472 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.702s user 0.012s sys 0.032s
W20260925 03:04:21.816811 25666 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.588s user 0.000s sys 0.016s
W20260925 03:04:21.963285 24544 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.580s user 0.006s sys 0.010s
W20260925 03:04:21.963608 24544 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.581s user 0.006s sys 0.010s
W20260925 03:04:21.957028 24473 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.664s user 0.004s sys 0.006s
W20260925 03:04:21.964340 24473 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.671s user 0.005s sys 0.006s
W20260925 03:04:21.958220 24439 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.591s user 0.000s sys 0.011s
W20260925 03:04:21.964932 24439 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.598s user 0.000s sys 0.012s
I20260925 03:04:21.967507 25700 raft_consensus.cc:482] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:21.959185 25584 consensus_queue.cc:1035] T c85f5d1d3f004cf5817bb594fd8e6469 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260925 03:04:21.885813 25710 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.631s user 0.011s sys 0.054s
W20260925 03:04:21.968181 25710 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.715s user 0.011s sys 0.054s
I20260925 03:04:21.967975 25700 raft_consensus.cc:504] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:21.997769 25584 raft_consensus.cc:482] T 7b8e0d260a884ecdb20f79de46c36b19 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:21.998855 25584 raft_consensus.cc:504] T 7b8e0d260a884ecdb20f79de46c36b19 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:22.002791 25584 leader_election.cc:288] T 7b8e0d260a884ecdb20f79de46c36b19 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
W20260925 03:04:21.913689 24619 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.746s user 0.010s sys 0.028s
W20260925 03:04:22.021572 24619 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.853s user 0.011s sys 0.029s
I20260925 03:04:22.008635 24588 leader_election.cc:302] T 43366a0a2f9d4344bf4e37ffd290ef8a P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
W20260925 03:04:21.885840 24469 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.763s user 0.000s sys 0.024s
W20260925 03:04:22.047883 24614 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.655s user 0.017s sys 0.021s
W20260925 03:04:22.049172 24614 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.656s user 0.017s sys 0.022s
W20260925 03:04:22.048864 24464 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.631s user 0.012s sys 0.010s
W20260925 03:04:22.052273 24464 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.634s user 0.012s sys 0.010s
W20260925 03:04:21.961160 25196 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.716s user 0.003s sys 0.028s
W20260925 03:04:22.047894 25666 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.826s user 0.000s sys 0.017s
W20260925 03:04:22.068838 24436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 109861us
I20260925 03:04:22.047516 25727 raft_consensus.cc:482] T 7993f9c3e2a44f408bbf5d1e395e2306 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:22.909242 24544 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 545) took 1791 ms. Trace:
W20260925 03:04:22.571022 24511 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.296s user 0.020s sys 0.057s
W20260925 03:04:22.050535 25648 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.614s user 0.006s sys 0.033s
I20260925 03:04:22.914312 24464 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 563) took 1850 ms. Trace:
W20260925 03:04:22.242549 24368 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.566s user 0.013s sys 0.022s
I20260925 03:04:22.904726 24615 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38570 (request call id 1123) took 1264 ms. Trace:
I20260925 03:04:22.337026 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6b6ec010caf4689b8c90449764f5cf9" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:22.902849 24471 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55560 (request call id 1099) took 1785 ms. Trace:
W20260925 03:04:22.385061 24586 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.225s user 0.014s sys 0.043s
W20260925 03:04:22.925467 24368 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.251s user 0.013s sys 0.064s
I20260925 03:04:22.183729 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15861cd28e354598a3c6bfc4337c34bb" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
W20260925 03:04:22.923903 24511 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.649s user 0.022s sys 0.089s
W20260925 03:04:22.927000 24586 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.767s user 0.031s sys 0.060s
W20260925 03:04:22.092415 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 83894us
W20260925 03:04:22.928350 24511 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 1.65355e+06us
I20260925 03:04:22.925894 24436 leader_election.cc:302] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:22.592324 23300 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260925 03:04:22.926760 24471 rpcz_store.cc:276] 0925 03:04:21.117428 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:21.117675 (+ 247us) service_pool.cc:227] Handling call
0925 03:04:21.117987 (+ 312us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:04:21.118326 (+ 339us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:21.118356 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:04:21.118774 (+ 418us) log.cc:842] Serialized 26 byte log entry
0925 03:04:21.297795 (+179021us) raft_consensus.cc:1631] Marking committed up to 1
0925 03:04:21.297847 (+ 52us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:21.297914 (+ 67us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:04:22.902562 (+1604648us) raft_consensus.cc:1671] finished
0925 03:04:22.902588 (+ 26us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:22.902729 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":606,"raft.run_cpu_time_us":103,"raft.run_wall_time_us":103,"thread_start_us":178622,"threads_started":1,"wal-append.queue_time_us":933105}
I20260925 03:04:22.928524 25702 raft_consensus.cc:3032] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 LEADER]: Stepping down as leader of term 2
I20260925 03:04:22.050659 25737 raft_consensus.cc:482] T c85f5d1d3f004cf5817bb594fd8e6469 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260925 03:04:22.363929 24512 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.695s user 0.000s sys 0.028s
W20260925 03:04:22.995106 24436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 69309.1us
W20260925 03:04:22.129732 24616 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.678s user 0.012s sys 0.012s
W20260925 03:04:22.377216 25755 log.cc:924] Time spent T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43: Append to log took a long time: real 0.495s user 0.000s sys 0.001s
I20260925 03:04:22.923424 24544 rpcz_store.cc:276] 0925 03:04:21.118283 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:21.203548 (+ 85265us) service_pool.cc:227] Handling call
0925 03:04:21.212607 (+ 9059us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:04:21.218255 (+ 5648us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:21.218286 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:04:21.231917 (+ 13631us) log.cc:842] Serialized 26 byte log entry
0925 03:04:21.382711 (+150794us) spinlock_profiling.cc:243] Waited 3.1 ms on lock 0x7b18000cdba0. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f437c4478b1 00007f437c454661 00007f437358a345 00007f437c3c1a39 00007f43735a78bb 00007f43735a5cbf 00007f43735a7e22 00007f43761515bb 00007f4376157063 00007f4376157411 00007f43764a45cf
0925 03:04:21.963964 (+581253us) raft_consensus.cc:1631] Marking committed up to 1
0925 03:04:21.964009 (+ 45us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:21.964079 (+ 70us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:04:22.908905 (+944826us) raft_consensus.cc:1671] finished
0925 03:04:22.908929 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:22.909166 (+ 237us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6734,"raft.queue_time_us":17742,"raft.run_cpu_time_us":150,"raft.run_wall_time_us":4284,"spinlock_wait_cycles":7144448,"thread_start_us":581047,"threads_started":1,"wal-append.queue_time_us":1671536}
W20260925 03:04:23.011526 24512 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.934s user 0.004s sys 0.060s
W20260925 03:04:22.191859 25658 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.767s user 0.043s sys 0.076s
I20260925 03:04:22.128751 25749 raft_consensus.cc:2726] T 43366a0a2f9d4344bf4e37ffd290ef8a P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260925 03:04:22.405948 25735 log.cc:924] Time spent T 7e49248643df4131a1c42e74244e05c2 P 1bc1f2e62ab44af995165bd5116a9d43: Append to log took a long time: real 0.331s user 0.001s sys 0.000s
W20260925 03:04:22.472138 25697 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.505s user 0.003s sys 0.034s
W20260925 03:04:22.742880 25746 log.cc:924] Time spent T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5: Append to log took a long time: real 0.135s user 0.000s sys 0.000s
W20260925 03:04:22.906884 24618 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.595s user 0.007s sys 0.002s
W20260925 03:04:22.835170 25193 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.261s user 0.021s sys 0.035s
I20260925 03:04:22.939670 24548 raft_consensus.cc:2454] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 2.
W20260925 03:04:22.820881 25748 log.cc:924] Time spent T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.050s user 0.000s sys 0.001s
I20260925 03:04:22.925843 24615 rpcz_store.cc:276] 0925 03:04:21.639876 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:21.640005 (+ 129us) service_pool.cc:227] Handling call
0925 03:04:21.640533 (+ 528us) raft_consensus.cc:1457] Updating replica for 2 ops
0925 03:04:21.641137 (+ 604us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:21.641163 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0925 03:04:21.642228 (+ 1065us) log.cc:842] Serialized 582 byte log entry
0925 03:04:22.106975 (+464747us) raft_consensus.cc:1631] Marking committed up to 2
0925 03:04:22.107014 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:22.107094 (+ 80us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:04:22.904390 (+797296us) raft_consensus.cc:1671] finished
0925 03:04:22.904413 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:22.904590 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":20030,"raft.run_cpu_time_us":165,"raft.run_wall_time_us":164,"thread_start_us":464328,"threads_started":1,"wal-append.queue_time_us":1259761}
W20260925 03:04:22.432637 24540 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.129s user 0.023s sys 0.014s
W20260925 03:04:22.105837 25647 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.895s user 0.012s sys 0.029s
W20260925 03:04:22.192428 25563 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.901s user 0.002s sys 0.027s
W20260925 03:04:22.786686 25714 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.977s user 0.000s sys 0.028s
W20260925 03:04:23.009142 24585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.4:43979, user_credentials={real_user=slave}} blocked reactor thread for 823787us
W20260925 03:04:22.932618 25648 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.496s user 0.081s sys 0.149s
W20260925 03:04:22.922350 24613 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.122s user 0.021s sys 0.064s
I20260925 03:04:22.267221 25678 raft_consensus.cc:482] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260925 03:04:22.359973 25711 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.082s user 0.014s sys 0.042s
I20260925 03:04:22.242116 25642 raft_consensus.cc:482] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:22.935822 25702 raft_consensus.cc:729] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:04:22.925468 24464 rpcz_store.cc:276] 0925 03:04:21.063502 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:21.063722 (+ 220us) service_pool.cc:227] Handling call
0925 03:04:21.070872 (+ 7150us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:04:21.328822 (+257950us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:21.328892 (+ 70us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:04:21.367323 (+ 38431us) log.cc:842] Serialized 26 byte log entry
0925 03:04:22.052715 (+685392us) raft_consensus.cc:1631] Marking committed up to 4
0925 03:04:22.052814 (+ 99us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:22.052883 (+ 69us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:04:22.914084 (+861201us) raft_consensus.cc:1671] finished
0925 03:04:22.914107 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:22.914248 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1301,"raft.queue_time_us":7286,"raft.run_cpu_time_us":169,"raft.run_wall_time_us":7209,"thread_start_us":882779,"threads_started":2,"wal-append.queue_time_us":1518063}
I20260925 03:04:22.910000 25700 leader_election.cc:288] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:22.382158 25560 raft_consensus.cc:482] T cc2df40db08649c9a75e90000b878559 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:23.059289 25763 raft_consensus.cc:482] T f22ddd7d1c444dc58f45a1fb0b3e02bc P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:22.360124 25664 raft_consensus.cc:482] T ac9357d144aa49789faaf31fe5e6bf89 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:22.084452 25754 raft_consensus.cc:482] T a4abe0ea757e4886be4945c5001e529a P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:22.903017 25785 raft_consensus.cc:482] T 04e068a0f80941dc9138cee063f53bab P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:22.545987 25516 raft_consensus.cc:482] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:22.145987 25649 raft_consensus.cc:482] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:22.913928 25784 raft_consensus.cc:482] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:23.099285 25784 raft_consensus.cc:504] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:23.100359 24619 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38564 (request call id 1113) took 2021 ms. Trace:
I20260925 03:04:22.097189 25685 raft_consensus.cc:482] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
W20260925 03:04:23.007696 24616 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.556s user 0.025s sys 0.032s
W20260925 03:04:23.013036 25658 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.596s user 0.074s sys 0.161s
I20260925 03:04:23.000540 24469 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55560 (request call id 1100) took 1875 ms. Trace:
I20260925 03:04:22.520964 25750 raft_consensus.cc:482] T b4ebd2b043024a548005c8b6b00b0449 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1bc1f2e62ab44af995165bd5116a9d43)
I20260925 03:04:23.078980 25666 raft_consensus.cc:482] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:23.046125 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f68804eb5a244d87bb395d50683c6012" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:23.102229 25787 raft_consensus.cc:482] T a4abe0ea757e4886be4945c5001e529a P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
W20260925 03:04:23.051918 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 94357us
W20260925 03:04:23.045262 25697 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.285s user 0.003s sys 0.049s
I20260925 03:04:22.931268 25727 raft_consensus.cc:504] T 7993f9c3e2a44f408bbf5d1e395e2306 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:04:23.058952 25563 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.768s user 0.005s sys 0.033s
I20260925 03:04:23.051465 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "550118bc1a0641ce9803bdbf43f2069f" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
W20260925 03:04:23.053805 25193 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.476s user 0.021s sys 0.045s
W20260925 03:04:23.049093 24618 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.745s user 0.007s sys 0.007s
I20260925 03:04:23.061940 24617 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38564 (request call id 1115) took 1961 ms. Trace:
W20260925 03:04:23.055711 24540 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.762s user 0.028s sys 0.029s
I20260925 03:04:23.053635 25710 raft_consensus.cc:482] T 1414dcfe53f64a78ae413531eb8ec117 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:23.070040 25196 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49794 (request call id 112) took 1996 ms. Trace:
W20260925 03:04:23.059136 25714 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.249s user 0.000s sys 0.033s
I20260925 03:04:23.056216 25738 raft_consensus.cc:2726] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260925 03:04:23.071727 25762 raft_consensus.cc:482] T 550118bc1a0641ce9803bdbf43f2069f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:23.051081 25740 raft_consensus.cc:482] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
W20260925 03:04:23.066418 24613 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.266s user 0.022s sys 0.069s
W20260925 03:04:23.060400 25647 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.850s user 0.030s sys 0.053s
I20260925 03:04:23.060926 25637 raft_consensus.cc:482] T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:22.816736 25641 raft_consensus.cc:482] T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:23.018038 25737 raft_consensus.cc:504] T c85f5d1d3f004cf5817bb594fd8e6469 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:04:23.070776 25711 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.817s user 0.020s sys 0.065s
W20260925 03:04:23.084803 24437 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.592s user 0.010s sys 0.077s
I20260925 03:04:23.057240 25636 consensus_queue.cc:1046] T 0b5df303cd6a4d59aca5c18bbbff24ed P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.020s
I20260925 03:04:23.074542 25751 raft_consensus.cc:482] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:23.071792 25728 raft_consensus.cc:482] T 74443a0a1dc34d88a2f42e3b722db405 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:23.086427 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e8af2641e124f9589c5ccfdac785f63" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:23.097535 25782 raft_consensus.cc:482] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:23.097213 25516 raft_consensus.cc:504] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:23.098284 25649 raft_consensus.cc:504] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:22.424823 25517 raft_consensus.cc:482] T 8ac466793d094325ace224708b2158ed P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:23.101572 24473 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55560 (request call id 1103) took 1979 ms. Trace:
W20260925 03:04:23.102028 25757 log.cc:924] Time spent T 37a157996e944db2906110d87ecb0c54 P 869b0a8575b54d08ae2817fad5dd42d5: Append to log took a long time: real 0.527s user 0.002s sys 0.007s
I20260925 03:04:23.079376 25648 raft_consensus.cc:482] T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:23.066247 25759 consensus_queue.cc:1035] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.047s
I20260925 03:04:22.329680 25741 raft_consensus.cc:482] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
W20260925 03:04:23.046815 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 117310us
I20260925 03:04:23.077338 25642 raft_consensus.cc:504] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:23.072002 25678 raft_consensus.cc:504] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:23.086369 25560 raft_consensus.cc:504] T cc2df40db08649c9a75e90000b878559 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:23.087653 25763 raft_consensus.cc:504] T f22ddd7d1c444dc58f45a1fb0b3e02bc P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:23.103015 24619 rpcz_store.cc:276] 0925 03:04:21.078609 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:21.078849 (+ 240us) service_pool.cc:227] Handling call
0925 03:04:21.092502 (+ 13653us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:04:21.093029 (+ 527us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:21.093056 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:04:21.098893 (+ 5837us) log.cc:842] Serialized 26 byte log entry
0925 03:04:22.075442 (+976549us) raft_consensus.cc:1631] Marking committed up to 1
0925 03:04:22.075486 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:22.075566 (+ 80us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:04:23.100103 (+1024537us) raft_consensus.cc:1671] finished
0925 03:04:23.100127 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:23.100290 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2380,"raft.queue_time_us":4779,"raft.run_cpu_time_us":167,"raft.run_wall_time_us":2877,"thread_start_us":857486,"threads_started":1,"wal-append.queue_time_us":1901731}
W20260925 03:04:23.199011 25163 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 270073us
W20260925 03:04:23.420876 24436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 417176us
W20260925 03:04:23.374265 25165 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.22.193.1:42187 timed out after 1.695s (SENT)
I20260925 03:04:23.108007 25199 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49794 (request call id 113) took 2014 ms. Trace:
W20260925 03:04:23.523061 24437 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.031s user 0.010s sys 0.089s
I20260925 03:04:23.522356 24617 rpcz_store.cc:276] 0925 03:04:21.100061 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:21.100330 (+ 269us) service_pool.cc:227] Handling call
0925 03:04:21.100661 (+ 331us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:04:21.103766 (+ 3105us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:21.103793 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:04:21.155675 (+ 51882us) log.cc:842] Serialized 26 byte log entry
0925 03:04:21.779003 (+623328us) raft_consensus.cc:1631] Marking committed up to 1
0925 03:04:21.779045 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:21.779122 (+ 77us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:04:23.061707 (+1282585us) raft_consensus.cc:1671] finished
0925 03:04:23.061729 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:23.061878 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4853,"raft.queue_time_us":3905,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":4014,"thread_start_us":607343,"threads_started":1,"wal-append.queue_time_us":1578924}
W20260925 03:04:23.508539 25770 log.cc:924] Time spent T 367277daf8d04778b3fa4ed94d71580d P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.303s user 0.000s sys 0.001s
I20260925 03:04:23.551144 24464 raft_consensus.cc:2454] T 9e8af2641e124f9589c5ccfdac785f63 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 2.
I20260925 03:04:23.553231 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5f9b503b0ab43999d582015a367441e" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:23.521802 24473 rpcz_store.cc:276] 0925 03:04:21.121869 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:21.180736 (+ 58867us) service_pool.cc:227] Handling call
0925 03:04:21.207991 (+ 27255us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:04:21.222651 (+ 14660us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:21.222683 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:04:21.235657 (+ 12974us) log.cc:842] Serialized 26 byte log entry
0925 03:04:21.965913 (+730256us) raft_consensus.cc:1631] Marking committed up to 1
0925 03:04:21.965954 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:21.966018 (+ 64us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:04:23.101373 (+1135355us) raft_consensus.cc:1671] finished
0925 03:04:23.101393 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:23.101518 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":10233,"raft.run_cpu_time_us":143,"raft.run_wall_time_us":8366,"thread_start_us":671637,"threads_started":1,"wal-append.queue_time_us":1805632}
I20260925 03:04:23.216192 25761 raft_consensus.cc:482] T 48ca6ccd98194cf2af7b5278d15bf4ea P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:23.540921 25199 rpcz_store.cc:276] 0925 03:04:21.093684 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:21.094162 (+ 478us) service_pool.cc:227] Handling call
0925 03:04:21.102933 (+ 8771us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:04:21.106926 (+ 3993us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:21.106955 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:04:21.158812 (+ 51857us) log.cc:842] Serialized 26 byte log entry
0925 03:04:21.949826 (+791014us) raft_consensus.cc:1631] Marking committed up to 4
0925 03:04:21.949868 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:21.949951 (+ 83us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:04:23.107779 (+1157828us) raft_consensus.cc:1671] finished
0925 03:04:23.107800 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:23.107946 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2004,"raft.run_cpu_time_us":150,"raft.run_wall_time_us":5849,"thread_start_us":731970,"threads_started":1,"wal-append.queue_time_us":1827652}
I20260925 03:04:23.297861 25749 raft_consensus.cc:482] T 100c113a9fd84cc8adf2ec7b33ea0093 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:23.101613 25784 leader_election.cc:288] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:23.106552 24616 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38564 (request call id 1111) took 2043 ms. Trace:
I20260925 03:04:23.093832 25664 raft_consensus.cc:504] T ac9357d144aa49789faaf31fe5e6bf89 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:23.412467 25733 raft_consensus.cc:482] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
W20260925 03:04:23.604502 24436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 78330.7us
I20260925 03:04:23.604067 24468 raft_consensus.cc:3037] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:23.594952 25711 raft_consensus.cc:482] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:23.543455 25740 raft_consensus.cc:504] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:23.602735 25563 raft_consensus.cc:482] T 04e068a0f80941dc9138cee063f53bab P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:23.620585 25516 leader_election.cc:288] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:23.539132 25787 raft_consensus.cc:504] T a4abe0ea757e4886be4945c5001e529a P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:23.623678 24613 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38564 (request call id 1114) took 2522 ms. Trace:
I20260925 03:04:23.623915 24472 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55560 (request call id 1102) took 2502 ms. Trace:
I20260925 03:04:23.543588 25751 raft_consensus.cc:504] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:23.094061 25785 raft_consensus.cc:504] T 04e068a0f80941dc9138cee063f53bab P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:23.624137 25749 raft_consensus.cc:504] T 100c113a9fd84cc8adf2ec7b33ea0093 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:23.538787 25710 raft_consensus.cc:504] T 1414dcfe53f64a78ae413531eb8ec117 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:23.539449 25517 raft_consensus.cc:504] T 8ac466793d094325ace224708b2158ed P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:23.625240 25763 leader_election.cc:288] T f22ddd7d1c444dc58f45a1fb0b3e02bc P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:23.602578 25762 raft_consensus.cc:504] T 550118bc1a0641ce9803bdbf43f2069f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:23.539386 25666 raft_consensus.cc:504] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:23.221179 25702 consensus_queue.cc:260] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:23.094149 25754 raft_consensus.cc:504] T a4abe0ea757e4886be4945c5001e529a P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:23.106863 25685 raft_consensus.cc:504] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:23.539229 25641 raft_consensus.cc:504] T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:23.626724 25678 leader_election.cc:288] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:23.628381 25642 leader_election.cc:288] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:23.188501 25750 raft_consensus.cc:504] T b4ebd2b043024a548005c8b6b00b0449 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:23.539160 25741 raft_consensus.cc:504] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:23.630266 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "293ef69caf514ac98510a68495f54eda" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:23.630787 25193 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49794 (request call id 114) took 2168 ms. Trace:
I20260925 03:04:23.525677 25196 rpcz_store.cc:276] 0925 03:04:21.073800 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:21.073948 (+ 148us) service_pool.cc:227] Handling call
0925 03:04:21.098846 (+ 24898us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:04:21.155402 (+ 56556us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:21.158485 (+ 3083us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:04:21.216331 (+ 57846us) log.cc:842] Serialized 26 byte log entry
0925 03:04:22.090208 (+873877us) raft_consensus.cc:1631] Marking committed up to 4
0925 03:04:22.100569 (+ 10361us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:22.158063 (+ 57494us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:04:23.067266 (+909203us) raft_consensus.cc:1671] finished
0925 03:04:23.067889 (+ 623us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:23.069952 (+ 2063us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2378,"raft.run_cpu_time_us":156,"raft.run_wall_time_us":5309,"thread_start_us":823148,"threads_started":1,"wal-append.queue_time_us":587775}
I20260925 03:04:23.630962 24539 raft_consensus.cc:1261] T c85f5d1d3f004cf5817bb594fd8e6469 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260925 03:04:23.543515 25782 raft_consensus.cc:504] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:23.631268 24541 raft_consensus.cc:1261] T 420163a6cdce412ebc21c25fe3835638 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260925 03:04:23.539541 25637 raft_consensus.cc:504] T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:23.539146 25648 raft_consensus.cc:504] T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:23.631779 25761 raft_consensus.cc:504] T 48ca6ccd98194cf2af7b5278d15bf4ea P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:23.632529 24473 raft_consensus.cc:1261] T f22ddd7d1c444dc58f45a1fb0b3e02bc P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260925 03:04:23.623621 24614 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38570 (request call id 1118) took 2793 ms. Trace:
I20260925 03:04:23.109336 25197 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48408 (request call id 104) took 2289 ms. Trace:
I20260925 03:04:23.114455 24469 rpcz_store.cc:276] 0925 03:04:21.120059 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:21.121668 (+ 1609us) service_pool.cc:227] Handling call
0925 03:04:21.122030 (+ 362us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:04:21.122429 (+ 399us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:21.122460 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:04:21.122924 (+ 464us) log.cc:842] Serialized 26 byte log entry
0925 03:04:22.055052 (+932128us) raft_consensus.cc:1631] Marking committed up to 1
0925 03:04:22.073933 (+ 18881us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:22.090689 (+ 16756us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:04:22.869679 (+778990us) raft_consensus.cc:1671] finished
0925 03:04:22.871195 (+ 1516us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:22.995573 (+124378us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2447,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":4713,"thread_start_us":927038,"threads_started":1,"wal-append.queue_time_us":1620663}
I20260925 03:04:23.640861 24616 rpcz_store.cc:276] 0925 03:04:21.062775 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:21.064324 (+ 1549us) service_pool.cc:227] Handling call
0925 03:04:21.447615 (+383291us) spinlock_profiling.cc:243] Waited 2.39 ms on lock 0x7b58001d3d98. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f43733ecf11 00007f4373570b7c 00007f437357073b 00007f437c460b09 00007f437b39c730 00007f437b46279b 00007f437b462821 00007f437b4262b0 00007f437b4117f9 00007f437b4112a2 00007f4374af7684
0925 03:04:21.447800 (+ 185us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:04:21.448237 (+ 437us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:21.448274 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:04:21.451338 (+ 3064us) log.cc:842] Serialized 26 byte log entry
0925 03:04:23.106057 (+1654719us) raft_consensus.cc:1631] Marking committed up to 1
0925 03:04:23.106097 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:23.106172 (+ 75us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:04:23.106230 (+ 58us) raft_consensus.cc:1671] finished
0925 03:04:23.106239 (+ 9us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:23.106489 (+ 250us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":133,"raft.run_cpu_time_us":119,"raft.run_wall_time_us":121,"spinlock_wait_cycles":5498752,"thread_start_us":1653165,"threads_started":1,"wal-append.queue_time_us":1453400}
I20260925 03:04:23.655121 25560 leader_election.cc:288] T cc2df40db08649c9a75e90000b878559 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:23.595090 25786 raft_consensus.cc:482] T 29637f5a4a504ce6ba29033ccd0cd716 P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:23.655732 25649 leader_election.cc:288] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:23.656734 25737 leader_election.cc:288] T c85f5d1d3f004cf5817bb594fd8e6469 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:23.656903 24470 raft_consensus.cc:1226] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 2. Current term is 3. Ops: []
I20260925 03:04:23.621632 25738 raft_consensus.cc:482] T 8e5819c15eac4544a57e5ee6c9d4f117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:23.526628 25728 raft_consensus.cc:504] T 74443a0a1dc34d88a2f42e3b722db405 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:23.625917 25789 raft_consensus.cc:482] T cc2df40db08649c9a75e90000b878559 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:23.658890 25727 leader_election.cc:288] T 7993f9c3e2a44f408bbf5d1e395e2306 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:23.661108 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60d45b7d467d4e1ab92367f6d7d52b15" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:23.664017 25796 raft_consensus.cc:482] T 100c113a9fd84cc8adf2ec7b33ea0093 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:23.681741 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abd5b8dc1b7e49e3aa9af900d73ff0e7" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:23.674692 24472 rpcz_store.cc:276] 0925 03:04:21.121338 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:21.170984 (+ 49646us) service_pool.cc:227] Handling call
0925 03:04:21.205166 (+ 34182us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:04:21.209609 (+ 4443us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:21.209640 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:04:21.223219 (+ 13579us) log.cc:842] Serialized 26 byte log entry
0925 03:04:21.950210 (+726991us) raft_consensus.cc:1631] Marking committed up to 1
0925 03:04:21.950247 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:21.950308 (+ 61us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:04:23.101500 (+1151192us) spinlock_profiling.cc:243] Waited 17.7 ms on lock 0x7b44000232cc. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f43733ecf11 00007f43734baa95 00007f43734bb4b2 00007f4376163511 00007f437615ee1a 00007f4376152bee 00007f4376151f36 00007f4376151b58 00007f437615b551 00007f437615b509 00007f437615b499
0925 03:04:23.623631 (+522131us) raft_consensus.cc:1671] finished
0925 03:04:23.623652 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:23.623841 (+ 189us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4010,"raft.queue_time_us":129,"raft.run_cpu_time_us":127,"raft.run_wall_time_us":5048,"spinlock_wait_cycles":40818560,"thread_start_us":708853,"threads_started":1,"wal-append.queue_time_us":1206764}
I20260925 03:04:23.746989 25647 raft_consensus.cc:482] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:23.749913 25738 raft_consensus.cc:504] T 8e5819c15eac4544a57e5ee6c9d4f117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:04:23.732004 25165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 357744us
I20260925 03:04:23.702235 25516 raft_consensus.cc:482] T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:23.735816 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b8e0d260a884ecdb20f79de46c36b19" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:23.751096 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:23.750554 25647 raft_consensus.cc:504] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:23.788080 24588 leader_election.cc:302] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:23.789400 24614 rpcz_store.cc:276] 0925 03:04:20.829757 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:20.866367 (+ 36610us) service_pool.cc:227] Handling call
0925 03:04:21.391578 (+525211us) spinlock_profiling.cc:243] Waited 1.87 ms on lock 0x7b58001d3d98. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f43733ecf11 00007f4373570851 00007f4373570702 00007f437c460b09 00007f437b39c730 00007f437b46279b 00007f437b462821 00007f437b4262b0 00007f437b4117f9 00007f437b4112a2 00007f4374af7684
0925 03:04:21.391796 (+ 218us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:04:21.392227 (+ 431us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:21.392256 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:04:21.392810 (+ 554us) log.cc:842] Serialized 26 byte log entry
0925 03:04:22.049566 (+656756us) raft_consensus.cc:1631] Marking committed up to 2
0925 03:04:22.049608 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:22.049680 (+ 72us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:04:23.620762 (+1571082us) raft_consensus.cc:1671] finished
0925 03:04:23.620781 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:23.623534 (+ 2753us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":23267,"raft.run_cpu_time_us":155,"raft.run_wall_time_us":155,"spinlock_wait_cycles":4292864,"thread_start_us":656352,"threads_started":1,"wal-append.queue_time_us":1684603}
I20260925 03:04:23.790194 25796 raft_consensus.cc:504] T 100c113a9fd84cc8adf2ec7b33ea0093 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:23.791620 25647 leader_election.cc:288] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:23.738786 25163 leader_election.cc:302] T e6b6ec010caf4689b8c90449764f5cf9 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:04:23.674964 25563 raft_consensus.cc:504] T 04e068a0f80941dc9138cee063f53bab P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:23.792429 25647 raft_consensus.cc:482] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:23.793174 25647 raft_consensus.cc:504] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:23.794132 25563 leader_election.cc:288] T 04e068a0f80941dc9138cee063f53bab P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:23.794659 25563 raft_consensus.cc:482] T 420163a6cdce412ebc21c25fe3835638 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:23.795074 25563 raft_consensus.cc:504] T 420163a6cdce412ebc21c25fe3835638 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:23.795197 25516 raft_consensus.cc:504] T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:23.795985 24469 raft_consensus.cc:1226] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 2. Current term is 3. Ops: []
I20260925 03:04:23.796342 25647 leader_election.cc:288] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:23.797096 25647 raft_consensus.cc:482] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:23.798243 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35992316cb8d4c028333388eff02ecde" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:23.797819 25647 raft_consensus.cc:504] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:04:23.741262 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 106667us
I20260925 03:04:23.750373 25727 raft_consensus.cc:482] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:23.789258 24544 raft_consensus.cc:2454] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 3.
I20260925 03:04:23.799232 25727 raft_consensus.cc:504] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:23.798485 25728 leader_election.cc:288] T 74443a0a1dc34d88a2f42e3b722db405 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:23.800981 25637 leader_election.cc:288] T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:23.801118 25647 leader_election.cc:288] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:23.801630 25727 leader_election.cc:288] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:23.801900 25647 raft_consensus.cc:2781] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260925 03:04:23.799544 25163 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 60848us
I20260925 03:04:23.802520 25647 raft_consensus.cc:482] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Starting leader election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:23.802809 24616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367277daf8d04778b3fa4ed94d71580d" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:23.803169 25647 raft_consensus.cc:3037] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:23.798576 25789 raft_consensus.cc:504] T cc2df40db08649c9a75e90000b878559 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:23.804627 24616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e5819c15eac4544a57e5ee6c9d4f117" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:23.799656 25697 raft_consensus.cc:2781] T e6b6ec010caf4689b8c90449764f5cf9 P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260925 03:04:23.797461 25516 leader_election.cc:288] T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:23.793083 25796 leader_election.cc:288] T 100c113a9fd84cc8adf2ec7b33ea0093 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:23.799079 24545 raft_consensus.cc:2454] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 2.
I20260925 03:04:23.803570 25727 consensus_queue.cc:1035] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
I20260925 03:04:23.804669 25761 leader_election.cc:288] T 48ca6ccd98194cf2af7b5278d15bf4ea P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:23.805095 25786 raft_consensus.cc:504] T 29637f5a4a504ce6ba29033ccd0cd716 P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:23.815554 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceab6b0fccd54e909a64f66fe81a9a1f" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:23.816262 24619 raft_consensus.cc:2454] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 2.
I20260925 03:04:23.815723 25647 raft_consensus.cc:504] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:23.805953 25789 leader_election.cc:288] T cc2df40db08649c9a75e90000b878559 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:23.818572 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f22ddd7d1c444dc58f45a1fb0b3e02bc" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:23.805894 25697 raft_consensus.cc:482] T e6b6ec010caf4689b8c90449764f5cf9 P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:23.818904 25647 leader_election.cc:288] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 3 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:23.819437 25697 raft_consensus.cc:3037] T e6b6ec010caf4689b8c90449764f5cf9 P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:23.820405 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71f461a6e2ff434993b78a2d077f0abd" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:23.823662 25647 consensus_queue.cc:1046] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260925 03:04:23.827142 25697 raft_consensus.cc:504] T e6b6ec010caf4689b8c90449764f5cf9 P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:23.810081 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7993f9c3e2a44f408bbf5d1e395e2306" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:23.792469 25648 leader_election.cc:288] T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:23.828545 25789 consensus_queue.cc:1046] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260925 03:04:23.829080 25761 raft_consensus.cc:3032] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [term 2 LEADER]: Stepping down as leader of term 2
I20260925 03:04:23.807430 25197 rpcz_store.cc:276] 0925 03:04:20.819715 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:20.871013 (+ 51298us) service_pool.cc:227] Handling call
0925 03:04:20.919172 (+ 48159us) raft_consensus.cc:1457] Updating replica for 2 ops
0925 03:04:20.919757 (+ 585us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:20.919793 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0925 03:04:21.018223 (+ 98430us) log.cc:842] Serialized 582 byte log entry
0925 03:04:21.312128 (+293905us) raft_consensus.cc:1631] Marking committed up to 2
0925 03:04:21.327395 (+ 15267us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:21.330358 (+ 2963us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:04:23.109007 (+1778649us) raft_consensus.cc:1671] finished
0925 03:04:23.109032 (+ 25us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:23.109167 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1373,"raft.run_cpu_time_us":106,"raft.run_wall_time_us":106,"thread_start_us":283581,"threads_started":1,"wal-append.queue_time_us":753854}
I20260925 03:04:23.816431 25563 leader_election.cc:288] T 420163a6cdce412ebc21c25fe3835638 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:23.671788 25711 raft_consensus.cc:504] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:23.738013 24468 raft_consensus.cc:1261] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260925 03:04:23.835647 25195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e068a0f80941dc9138cee063f53bab" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:23.725131 25192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71f461a6e2ff434993b78a2d077f0abd" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:23.837217 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b5df303cd6a4d59aca5c18bbbff24ed" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:23.839425 25807 raft_consensus.cc:482] T 0b5df303cd6a4d59aca5c18bbbff24ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:23.840183 25785 leader_election.cc:288] T 04e068a0f80941dc9138cee063f53bab P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:23.843660 25750 leader_election.cc:288] T b4ebd2b043024a548005c8b6b00b0449 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:23.912882 25698 raft_consensus.cc:482] T f22ddd7d1c444dc58f45a1fb0b3e02bc P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:23.742960 25792 raft_consensus.cc:482] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:23.668996 25733 raft_consensus.cc:504] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:23.733757 25702 raft_consensus.cc:3037] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:23.890010 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e068a0f80941dc9138cee063f53bab" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:23.717252 25788 raft_consensus.cc:482] T e92f377a79354a6ebb4eec67303c13e3 P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:23.864662 25728 raft_consensus.cc:482] T 37a157996e944db2906110d87ecb0c54 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:23.911298 24473 raft_consensus.cc:2454] T 7993f9c3e2a44f408bbf5d1e395e2306 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 1.
I20260925 03:04:23.897455 25700 raft_consensus.cc:482] T 4e5332a5d329493991dff994e70ea92c P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:23.914114 24543 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33068 (request call id 53) took 1665 ms. Trace:
I20260925 03:04:23.740525 25636 raft_consensus.cc:3032] T 0b5df303cd6a4d59aca5c18bbbff24ed P eeab93abfdb141a895002ec9f312296c [term 2 LEADER]: Stepping down as leader of term 2
W20260925 03:04:23.965667 24585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.4:43979, user_credentials={real_user=slave}} blocked reactor thread for 52517.4us
I20260925 03:04:23.974503 25765 raft_consensus.cc:482] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:23.878047 25799 raft_consensus.cc:482] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:24.062477 24548 raft_consensus.cc:3037] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:23.832579 25763 consensus_queue.cc:1046] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.084s
I20260925 03:04:23.916540 25784 raft_consensus.cc:482] T b1e85248d3b04b979dd080caeeb02fdb P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:23.846568 25749 leader_election.cc:288] T 100c113a9fd84cc8adf2ec7b33ea0093 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
W20260925 03:04:24.002074 24436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 286315us
I20260925 03:04:23.879251 25741 leader_election.cc:288] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:23.882488 25737 raft_consensus.cc:482] T 25102ba08e4e49388f6176badcd10a25 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:24.006022 24511 leader_election.cc:302] T 3d89396576884f7c8901dba54f783947 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:23.738730 25678 raft_consensus.cc:482] T 11bb46bce3564841b7674e6a693bd2e6 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:23.741832 25193 rpcz_store.cc:276] 0925 03:04:21.462125 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:21.462529 (+ 404us) service_pool.cc:227] Handling call
0925 03:04:21.463211 (+ 682us) raft_consensus.cc:1457] Updating replica for 2 ops
0925 03:04:21.463828 (+ 617us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:21.463857 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0925 03:04:21.465995 (+ 2138us) log.cc:842] Serialized 582 byte log entry
0925 03:04:21.573529 (+107534us) spinlock_profiling.cc:243] Waited 2.44 ms on lock 0x7b180027eec0. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f437c4478b1 00007f437c454661 00007f437358a345 00007f437c3c1a39 00007f43735a78bb 00007f43735a5cbf 00007f43735a7e22 00007f43761515bb 00007f4376157063 00007f4376157411 00007f43764a45cf
0925 03:04:23.543495 (+1969966us) raft_consensus.cc:1631] Marking committed up to 2
0925 03:04:23.552923 (+ 9428us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:23.583484 (+ 30561us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:04:23.601108 (+ 17624us) raft_consensus.cc:1671] finished
0925 03:04:23.604159 (+ 3051us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:23.630713 (+ 26554us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1571,"raft.run_cpu_time_us":112,"raft.run_wall_time_us":113,"spinlock_wait_cycles":5605120,"thread_start_us":1948218,"threads_started":1,"wal-append.queue_time_us":1436694}
I20260925 03:04:23.929203 25761 raft_consensus.cc:729] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:24.057003 25636 raft_consensus.cc:729] T 0b5df303cd6a4d59aca5c18bbbff24ed P eeab93abfdb141a895002ec9f312296c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:23.988873 25647 raft_consensus.cc:3032] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [term 2 LEADER]: Stepping down as leader of term 2
I20260925 03:04:24.049261 24543 rpcz_store.cc:276] 0925 03:04:22.248410 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:22.248571 (+ 161us) service_pool.cc:227] Handling call
0925 03:04:22.920548 (+671977us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:23.844149 (+923601us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:23.844178 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:23.844219 (+ 41us) raft_consensus.cc:1631] Marking committed up to 5
0925 03:04:23.844246 (+ 27us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:23.913890 (+ 69644us) spinlock_profiling.cc:243] Waited 923 ms on lock 0x7b540034add4. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d0f51 00007f43764cf99e 00007f437b4112e9 00007f4374af7684 00007f4374af7612 00007f4374af7541 00007f4374af74bc 00007f4374af6772 00007f43748fa94c 00007f43748f9d86
0925 03:04:23.913934 (+ 44us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:23.914050 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2123641216}
I20260925 03:04:24.072703 24614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abd5b8dc1b7e49e3aa9af900d73ff0e7" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
W20260925 03:04:24.049726 25806 log.cc:924] Time spent T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5: Append to log took a long time: real 0.171s user 0.000s sys 0.001s
I20260925 03:04:24.037034 25664 leader_election.cc:288] T ac9357d144aa49789faaf31fe5e6bf89 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:24.035938 25196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e54ca1757ee4e42956a9cebbeca4f62" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:24.085309 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e5819c15eac4544a57e5ee6c9d4f117" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:23.972540 24618 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34546 (request call id 38) took 2162 ms. Trace:
I20260925 03:04:24.091974 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e068a0f80941dc9138cee063f53bab" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:24.121942 24619 raft_consensus.cc:2454] T 04e068a0f80941dc9138cee063f53bab P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 1.
I20260925 03:04:24.033960 25698 raft_consensus.cc:504] T f22ddd7d1c444dc58f45a1fb0b3e02bc P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:24.084568 25737 raft_consensus.cc:504] T 25102ba08e4e49388f6176badcd10a25 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:24.098367 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b8e0d260a884ecdb20f79de46c36b19" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
W20260925 03:04:24.103915 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 130463us
I20260925 03:04:24.069726 24465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367277daf8d04778b3fa4ed94d71580d" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:24.111879 25195 raft_consensus.cc:1226] T e6b6ec010caf4689b8c90449764f5cf9 P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 2. Current term is 3. Ops: []
I20260925 03:04:23.681444 24613 rpcz_store.cc:276] 0925 03:04:21.093375 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:21.094311 (+ 936us) service_pool.cc:227] Handling call
0925 03:04:21.109375 (+ 15064us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:04:21.799622 (+690247us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:21.799672 (+ 50us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:04:21.800188 (+ 516us) log.cc:842] Serialized 26 byte log entry
0925 03:04:23.543681 (+1743493us) raft_consensus.cc:1631] Marking committed up to 1
0925 03:04:23.551773 (+ 8092us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:23.583598 (+ 31825us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:04:23.594504 (+ 10906us) raft_consensus.cc:1671] finished
0925 03:04:23.602424 (+ 7920us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:23.615264 (+ 12840us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":915,"raft.queue_time_us":70168,"raft.run_cpu_time_us":155,"raft.run_wall_time_us":4469,"thread_start_us":2316901,"threads_started":2,"wal-append.queue_time_us":1110907}
I20260925 03:04:24.098057 25799 raft_consensus.cc:504] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:04:23.741580 25798 consensus_queue.cc:1159] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "eeab93abfdb141a895002ec9f312296c" responder_term: 3 status { last_received { term: 2 index: 5 } last_committed_idx: 5 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 2. Current term is 3. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260925 03:04:24.049463 25788 raft_consensus.cc:504] T e92f377a79354a6ebb4eec67303c13e3 P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:24.077446 25784 raft_consensus.cc:504] T b1e85248d3b04b979dd080caeeb02fdb P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:24.119091 25751 leader_election.cc:288] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:24.115767 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f22ddd7d1c444dc58f45a1fb0b3e02bc" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:24.097108 25793 raft_consensus.cc:482] T 420163a6cdce412ebc21c25fe3835638 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:24.117409 25198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5aec38e66a947c681e6c2a72b05e6ca" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
W20260925 03:04:24.123478 24511 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 117545us
I20260925 03:04:24.057169 25728 raft_consensus.cc:504] T 37a157996e944db2906110d87ecb0c54 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:24.119975 25738 leader_election.cc:288] T 8e5819c15eac4544a57e5ee6c9d4f117 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:24.111881 25649 consensus_queue.cc:1046] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.049s
I20260925 03:04:24.087915 25765 raft_consensus.cc:504] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:24.097126 25697 leader_election.cc:288] T e6b6ec010caf4689b8c90449764f5cf9 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 3 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:24.100306 25647 raft_consensus.cc:729] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:24.073915 25700 raft_consensus.cc:504] T 4e5332a5d329493991dff994e70ea92c P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:24.146889 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceab6b0fccd54e909a64f66fe81a9a1f" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:24.104662 24616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5aec38e66a947c681e6c2a72b05e6ca" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:24.149107 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4abe0ea757e4886be4945c5001e529a" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:24.155534 24467 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 572) took 1218 ms. Trace:
I20260925 03:04:24.090058 25678 raft_consensus.cc:504] T 11bb46bce3564841b7674e6a693bd2e6 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:24.125923 25763 raft_consensus.cc:2726] T 3d89396576884f7c8901dba54f783947 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 03:04:24.070676 25808 raft_consensus.cc:482] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:24.042529 25807 raft_consensus.cc:504] T 0b5df303cd6a4d59aca5c18bbbff24ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:24.169816 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60d45b7d467d4e1ab92367f6d7d52b15" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:24.214226 25737 leader_election.cc:288] T 25102ba08e4e49388f6176badcd10a25 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:24.214411 25782 leader_election.cc:288] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:24.125377 25636 consensus_queue.cc:260] T 0b5df303cd6a4d59aca5c18bbbff24ed P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:24.171110 25198 raft_consensus.cc:2454] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 3.
I20260925 03:04:24.162262 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ac466793d094325ace224708b2158ed" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:24.134357 24437 leader_election.cc:302] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:04:24.169610 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6b6ec010caf4689b8c90449764f5cf9" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:24.140641 25761 consensus_queue.cc:260] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:24.068789 25792 raft_consensus.cc:504] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:24.239760 24467 rpcz_store.cc:276] 0925 03:04:22.937612 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:22.937736 (+ 124us) service_pool.cc:227] Handling call
0925 03:04:22.938299 (+ 563us) raft_consensus.cc:1457] Updating replica for 2 ops
0925 03:04:23.418984 (+480685us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:23.419018 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0925 03:04:23.494783 (+ 75765us) log.cc:842] Serialized 582 byte log entry
0925 03:04:23.670211 (+175428us) raft_consensus.cc:1631] Marking committed up to 2
0925 03:04:23.670281 (+ 70us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:23.670369 (+ 88us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:04:24.155316 (+484947us) raft_consensus.cc:1671] finished
0925 03:04:24.155336 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:24.155472 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":79350,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":12797,"thread_start_us":494232,"threads_started":2,"wal-append.queue_time_us":184646}
W20260925 03:04:24.269088 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 134824us
I20260925 03:04:24.270509 24437 leader_election.cc:302] T b627ecb4a5384ef392aa695e06599c39 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:24.178752 25754 leader_election.cc:288] T a4abe0ea757e4886be4945c5001e529a P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:24.155913 25666 leader_election.cc:288] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:24.154279 25787 leader_election.cc:288] T a4abe0ea757e4886be4945c5001e529a P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:24.207418 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5332a5d329493991dff994e70ea92c" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:24.160557 25762 leader_election.cc:288] T 550118bc1a0641ce9803bdbf43f2069f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:24.276525 24614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5aec38e66a947c681e6c2a72b05e6ca" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:24.290859 24542 raft_consensus.cc:3037] T 0b5df303cd6a4d59aca5c18bbbff24ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:24.160267 25517 leader_election.cc:288] T 8ac466793d094325ace224708b2158ed P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:24.290815 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9357d144aa49789faaf31fe5e6bf89" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:24.225230 25789 raft_consensus.cc:482] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:24.242431 24466 raft_consensus.cc:2454] T f22ddd7d1c444dc58f45a1fb0b3e02bc P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 3.
I20260925 03:04:24.253199 25685 leader_election.cc:288] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:24.246027 25765 leader_election.cc:288] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:24.300633 25198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4abe0ea757e4886be4945c5001e529a" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:24.304117 24467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b8e0d260a884ecdb20f79de46c36b19" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:24.313326 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ac466793d094325ace224708b2158ed" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:24.287469 25782 raft_consensus.cc:2781] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 03:04:24.342468 24615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4abe0ea757e4886be4945c5001e529a" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:24.345672 24588 leader_election.cc:302] T 7993f9c3e2a44f408bbf5d1e395e2306 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:24.269134 25711 leader_election.cc:288] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:24.293833 25636 raft_consensus.cc:3037] T 0b5df303cd6a4d59aca5c18bbbff24ed P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:24.190737 25793 raft_consensus.cc:504] T 420163a6cdce412ebc21c25fe3835638 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:24.303387 24465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc2df40db08649c9a75e90000b878559" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
W20260925 03:04:24.383960 24586 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.861s user 0.016s sys 0.014s
I20260925 03:04:24.304117 25761 raft_consensus.cc:3037] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:24.399027 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b030d84c670b48d5bff337ccc0c77626" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:24.398201 24542 raft_consensus.cc:2454] T 0b5df303cd6a4d59aca5c18bbbff24ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 3.
W20260925 03:04:24.397629 24586 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.875s user 0.016s sys 0.015s
W20260925 03:04:24.405597 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 882822us
I20260925 03:04:24.306037 24469 raft_consensus.cc:2454] T 4e5332a5d329493991dff994e70ea92c P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 3.
I20260925 03:04:24.349875 25754 raft_consensus.cc:482] T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:24.359818 25765 raft_consensus.cc:482] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:24.344239 25789 raft_consensus.cc:504] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:24.366343 25751 raft_consensus.cc:2781] T 7993f9c3e2a44f408bbf5d1e395e2306 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 03:04:24.418751 24467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abd5b8dc1b7e49e3aa9af900d73ff0e7" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:24.342497 25162 leader_election.cc:302] T 04e068a0f80941dc9138cee063f53bab P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, eeab93abfdb141a895002ec9f312296c; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:24.414984 25517 raft_consensus.cc:482] T 7e49248643df4131a1c42e74244e05c2 P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:24.304750 25647 consensus_queue.cc:260] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:24.404995 25782 raft_consensus.cc:686] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 LEADER]: Becoming Leader. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:04:24.406811 25784 leader_election.cc:288] T b1e85248d3b04b979dd080caeeb02fdb P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:24.258895 25808 raft_consensus.cc:504] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:24.497000 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c85f5d1d3f004cf5817bb594fd8e6469" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:24.504551 25698 leader_election.cc:288] T f22ddd7d1c444dc58f45a1fb0b3e02bc P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:24.523847 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48ca6ccd98194cf2af7b5278d15bf4ea" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:24.534051 25829 raft_consensus.cc:482] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:24.534300 25678 leader_election.cc:288] T 11bb46bce3564841b7674e6a693bd2e6 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:24.533664 25700 leader_election.cc:288] T 4e5332a5d329493991dff994e70ea92c P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:24.474418 25765 raft_consensus.cc:504] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:24.504245 25727 raft_consensus.cc:482] T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:24.534683 25789 leader_election.cc:288] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:24.491348 25751 raft_consensus.cc:482] T 7993f9c3e2a44f408bbf5d1e395e2306 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting leader election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:24.534611 25819 raft_consensus.cc:2932] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [term 3 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:24.534855 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc2df40db08649c9a75e90000b878559" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
W20260925 03:04:24.625665 24436 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.552s user 0.000s sys 0.032s
W20260925 03:04:24.700955 24436 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.627s user 0.001s sys 0.037s
W20260925 03:04:24.623662 24511 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 353003us
I20260925 03:04:24.547802 24467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "100c113a9fd84cc8adf2ec7b33ea0093" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:24.121583 24618 rpcz_store.cc:276] 0925 03:04:21.810473 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:21.893393 (+ 82920us) service_pool.cc:227] Handling call
0925 03:04:22.073987 (+180594us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:22.148345 (+ 74358us) raft_consensus.cc:1520] Early marking committed up to index 5
0925 03:04:23.550480 (+1402135us) log.cc:842] Serialized 14 byte log entry
0925 03:04:23.912066 (+361586us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:23.912141 (+ 75us) raft_consensus.cc:1631] Marking committed up to 5
0925 03:04:23.912191 (+ 50us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:23.912273 (+ 82us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:23.946447 (+ 34174us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11357,"prepare.queue_time_us":733799,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":43,"raft.queue_time_us":35382,"raft.run_cpu_time_us":84,"raft.run_wall_time_us":89,"thread_start_us":1494717,"threads_started":3}
I20260925 03:04:24.733080 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceab6b0fccd54e909a64f66fe81a9a1f" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
W20260925 03:04:24.739104 24548 consensus_meta.cc:287] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent flushing consensus metadata: real 0.669s user 0.008s sys 0.003s
W20260925 03:04:24.670745 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 371119us
I20260925 03:04:24.523415 25785 raft_consensus.cc:2781] T 04e068a0f80941dc9138cee063f53bab P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 03:04:24.619854 24621 raft_consensus.cc:2396] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260925 03:04:24.741073 24548 raft_consensus.cc:2454] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 3.
I20260925 03:04:24.408675 25733 leader_election.cc:288] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
W20260925 03:04:24.699662 25815 log.cc:924] Time spent T 71f461a6e2ff434993b78a2d077f0abd P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.092s user 0.000s sys 0.001s
I20260925 03:04:24.747926 24511 leader_election.cc:302] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
W20260925 03:04:24.808230 24511 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 60409.3us
I20260925 03:04:24.734999 25751 raft_consensus.cc:3037] T 7993f9c3e2a44f408bbf5d1e395e2306 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:24.522619 25710 leader_election.cc:288] T 1414dcfe53f64a78ae413531eb8ec117 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:24.529317 25711 raft_consensus.cc:482] T 442d4a29b1d743599f4b6b4871e26668 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
W20260925 03:04:24.803800 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 46052.5us
W20260925 03:04:24.806289 25826 log.cc:924] Time spent T 367277daf8d04778b3fa4ed94d71580d P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.421s user 0.000s sys 0.001s
I20260925 03:04:24.624184 24465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4ebd2b043024a548005c8b6b00b0449" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:24.766263 24467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e8af2641e124f9589c5ccfdac785f63" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:24.862941 24467 raft_consensus.cc:3037] T 9e8af2641e124f9589c5ccfdac785f63 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Advancing to term 3
W20260925 03:04:24.532083 25162 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 234546us
I20260925 03:04:24.700089 25829 raft_consensus.cc:504] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
W20260925 03:04:24.857323 25838 log.cc:924] Time spent T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.358s user 0.004s sys 0.000s
I20260925 03:04:24.597524 25790 raft_consensus.cc:482] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:24.567317 25728 leader_election.cc:288] T 37a157996e944db2906110d87ecb0c54 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:24.761202 25664 raft_consensus.cc:482] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:24.512187 25517 raft_consensus.cc:504] T 7e49248643df4131a1c42e74244e05c2 P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:24.763440 25700 raft_consensus.cc:482] T b4ebd2b043024a548005c8b6b00b0449 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1bc1f2e62ab44af995165bd5116a9d43)
I20260925 03:04:24.760879 25785 raft_consensus.cc:482] T 04e068a0f80941dc9138cee063f53bab P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:24.631460 25799 leader_election.cc:288] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:24.733701 25788 leader_election.cc:288] T e92f377a79354a6ebb4eec67303c13e3 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:24.867846 25789 raft_consensus.cc:482] T 71f461a6e2ff434993b78a2d077f0abd P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1bc1f2e62ab44af995165bd5116a9d43)
I20260925 03:04:24.521003 25754 raft_consensus.cc:504] T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:24.598694 25795 consensus_queue.cc:1035] T 37a157996e944db2906110d87ecb0c54 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.597s
I20260925 03:04:24.847756 25733 raft_consensus.cc:482] T 21c43af5e6844fdaacc8c625fa01fd8d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:24.738257 25727 raft_consensus.cc:504] T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:04:24.928879 25824 log.cc:924] Time spent T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.221s user 0.001s sys 0.000s
I20260925 03:04:24.929425 25812 raft_consensus.cc:482] T 21c43af5e6844fdaacc8c625fa01fd8d P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:24.867240 25666 raft_consensus.cc:482] T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:24.889848 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06f4a7eaf6534e4986c9fd35911038e4" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:24.816437 25792 leader_election.cc:288] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:24.974285 25765 leader_election.cc:288] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:24.973322 25198 raft_consensus.cc:2932] T 29637f5a4a504ce6ba29033ccd0cd716 P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:24.960453 25799 raft_consensus.cc:2781] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260925 03:04:24.976013 25797 raft_consensus.cc:482] T c88cbd39e01f44628a094af516be2833 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:24.977147 24464 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 581) took 1358 ms. Trace:
I20260925 03:04:25.038693 24464 rpcz_store.cc:276] 0925 03:04:23.619003 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:23.619130 (+ 127us) service_pool.cc:227] Handling call
0925 03:04:23.657581 (+ 38451us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:23.806538 (+148957us) raft_consensus.cc:1520] Early marking committed up to index 2
0925 03:04:23.828415 (+ 21877us) log.cc:842] Serialized 14 byte log entry
0925 03:04:24.141831 (+313416us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:24.141881 (+ 50us) raft_consensus.cc:1631] Marking committed up to 2
0925 03:04:24.141921 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:24.976880 (+834959us) spinlock_profiling.cc:243] Waited 149 ms on lock 0x7b54001abdd4. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d0f51 00007f43764cf99e 00007f437b4112e9 00007f4374af7684 00007f4374af7612 00007f4374af7541 00007f4374af74bc 00007f4374af6772 00007f43748fa94c 00007f43748f9d86
0925 03:04:24.976933 (+ 53us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:24.977079 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":553385,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":37,"raft.queue_time_us":16699,"raft.run_cpu_time_us":192,"raft.run_wall_time_us":9586,"spinlock_wait_cycles":342118144,"thread_start_us":325173,"threads_started":2}
I20260925 03:04:24.980300 24618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1e85248d3b04b979dd080caeeb02fdb" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:24.925923 25785 raft_consensus.cc:3037] T 04e068a0f80941dc9138cee063f53bab P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:24.984647 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ac285175464fbda67211d9574db415" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:24.884429 25711 raft_consensus.cc:504] T 442d4a29b1d743599f4b6b4871e26668 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:25.008865 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e725b82985746b39c0cc8f671c95353" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:24.920348 25700 raft_consensus.cc:504] T b4ebd2b043024a548005c8b6b00b0449 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:25.006765 24436 leader_election.cc:302] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5; no voters:
I20260925 03:04:25.021915 24367 catalog_manager.cc:5495] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "eeab93abfdb141a895002ec9f312296c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:25.027559 25808 leader_election.cc:288] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:24.961402 25801 raft_consensus.cc:482] T 3d89396576884f7c8901dba54f783947 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:24.923810 25664 raft_consensus.cc:504] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:25.035847 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e92f377a79354a6ebb4eec67303c13e3" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:24.520323 25685 raft_consensus.cc:2932] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:24.993906 25733 raft_consensus.cc:504] T 21c43af5e6844fdaacc8c625fa01fd8d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:04:25.055747 24436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.4:43979, user_credentials={real_user=slave}} blocked reactor thread for 49101us
I20260925 03:04:25.065311 25809 raft_consensus.cc:482] T 0be4e0aa984f4f2b81a58e382e434ad5 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:25.014693 25807 leader_election.cc:288] T 0b5df303cd6a4d59aca5c18bbbff24ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:25.067935 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "100c113a9fd84cc8adf2ec7b33ea0093" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:25.035370 25799 raft_consensus.cc:482] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:25.056552 24367 catalog_manager.cc:5495] T 29637f5a4a504ce6ba29033ccd0cd716 P 1bc1f2e62ab44af995165bd5116a9d43 reported cstate change: term changed from 1 to 3, now has a pending config: VOTER 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1), VOTER a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2), VOTER eeab93abfdb141a895002ec9f312296c (127.22.193.3), VOTER 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4). New cstate: current_term: 3 leader_uuid: "eeab93abfdb141a895002ec9f312296c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } } } pending_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:25.020838 25666 raft_consensus.cc:504] T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:25.030179 24466 raft_consensus.cc:2454] T 06f4a7eaf6534e4986c9fd35911038e4 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 3.
I20260925 03:04:25.079329 24617 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38570 (request call id 1140) took 1006 ms. Trace:
I20260925 03:04:25.090104 24617 rpcz_store.cc:276] 0925 03:04:24.072582 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:24.073179 (+ 597us) service_pool.cc:227] Handling call
0925 03:04:24.073473 (+ 294us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:04:24.075352 (+ 1879us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:24.075385 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:04:24.098270 (+ 22885us) log.cc:842] Serialized 26 byte log entry
0925 03:04:24.291523 (+193253us) raft_consensus.cc:1631] Marking committed up to 1
0925 03:04:24.291560 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:24.291622 (+ 62us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:04:25.079124 (+787502us) raft_consensus.cc:1671] finished
0925 03:04:25.079143 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:25.079271 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":15926,"raft.queue_time_us":24909,"raft.run_cpu_time_us":128,"raft.run_wall_time_us":129,"thread_start_us":153166,"threads_started":1,"wal-append.queue_time_us":812107}
I20260925 03:04:25.086894 24367 catalog_manager.cc:5495] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43 reported cstate change: term changed from 2 to 3, leader changed from 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) to eeab93abfdb141a895002ec9f312296c (127.22.193.3). New cstate: current_term: 3 leader_uuid: "eeab93abfdb141a895002ec9f312296c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:24.966418 25789 raft_consensus.cc:504] T 71f461a6e2ff434993b78a2d077f0abd P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:25.098366 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "420163a6cdce412ebc21c25fe3835638" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:24.522850 25782 consensus_queue.cc:237] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:25.100885 24464 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 586) took 1038 ms. Trace:
I20260925 03:04:25.019541 25791 consensus_queue.cc:1046] T e6b6ec010caf4689b8c90449764f5cf9 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.048s
I20260925 03:04:25.103549 24464 rpcz_store.cc:276] 0925 03:04:24.062090 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:25.088168 (+1026078us) service_pool.cc:227] Handling call
0925 03:04:25.100819 (+ 12651us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:25.081035 25799 raft_consensus.cc:3037] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:25.080592 24465 raft_consensus.cc:2454] T b4ebd2b043024a548005c8b6b00b0449 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 2.
I20260925 03:04:25.035127 25812 raft_consensus.cc:504] T 21c43af5e6844fdaacc8c625fa01fd8d P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:25.112038 25812 leader_election.cc:288] T 21c43af5e6844fdaacc8c625fa01fd8d P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:25.112994 25825 raft_consensus.cc:482] T 367277daf8d04778b3fa4ed94d71580d P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:25.097918 24466 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 588) took 1012 ms. Trace:
I20260925 03:04:25.087822 25809 raft_consensus.cc:504] T 0be4e0aa984f4f2b81a58e382e434ad5 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:25.121508 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6b6ec010caf4689b8c90449764f5cf9" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:25.084853 25801 raft_consensus.cc:504] T 3d89396576884f7c8901dba54f783947 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:25.113626 25825 raft_consensus.cc:504] T 367277daf8d04778b3fa4ed94d71580d P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } }
I20260925 03:04:24.968164 25790 raft_consensus.cc:504] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:25.079217 25797 raft_consensus.cc:504] T c88cbd39e01f44628a094af516be2833 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:25.138710 24467 raft_consensus.cc:2454] T 9e8af2641e124f9589c5ccfdac785f63 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 3.
I20260925 03:04:25.191498 24467 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 594) took 1045 ms. Trace:
I20260925 03:04:25.191972 24467 rpcz_store.cc:276] 0925 03:04:24.146527 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:24.734433 (+587906us) service_pool.cc:227] Handling call
0925 03:04:25.191301 (+456868us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:25.139163 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5aec38e66a947c681e6c2a72b05e6ca" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:25.203807 24467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e54ca1757ee4e42956a9cebbeca4f62" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:25.210184 24467 raft_consensus.cc:2454] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 3.
I20260925 03:04:25.125741 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e54ca1757ee4e42956a9cebbeca4f62" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:25.123374 25754 leader_election.cc:288] T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
W20260925 03:04:25.205781 25837 log.cc:924] Time spent T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5: Append to log took a long time: real 0.089s user 0.001s sys 0.000s
I20260925 03:04:25.143018 25829 leader_election.cc:288] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:25.226408 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6b6ec010caf4689b8c90449764f5cf9" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:25.118129 24466 rpcz_store.cc:276] 0925 03:04:24.082445 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:24.737968 (+655523us) service_pool.cc:227] Handling call
0925 03:04:25.095055 (+357087us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:25.161303 25727 leader_election.cc:288] T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
W20260925 03:04:25.222611 25197 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.919s user 0.011s sys 0.036s
I20260925 03:04:25.240056 25825 leader_election.cc:288] T 367277daf8d04778b3fa4ed94d71580d P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:25.240669 25845 raft_consensus.cc:3032] T e6b6ec010caf4689b8c90449764f5cf9 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 LEADER]: Stepping down as leader of term 2
I20260925 03:04:25.241544 25809 leader_election.cc:288] T 0be4e0aa984f4f2b81a58e382e434ad5 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:25.231778 25517 leader_election.cc:288] T 7e49248643df4131a1c42e74244e05c2 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:25.299966 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:25.234377 25711 leader_election.cc:288] T 442d4a29b1d743599f4b6b4871e26668 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:25.311584 24617 raft_consensus.cc:2454] T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 3.
W20260925 03:04:25.248647 25197 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.952s user 0.012s sys 0.036s
I20260925 03:04:25.249850 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c23337a7cb74184a0e57b7095cb6a11" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:25.315022 24464 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1118) took 1227 ms. Trace:
I20260925 03:04:25.315460 24464 rpcz_store.cc:276] 0925 03:04:24.087878 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:25.232671 (+1144793us) service_pool.cc:227] Handling call
0925 03:04:25.314959 (+ 82288us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:25.320938 25664 leader_election.cc:288] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:25.234766 25700 leader_election.cc:288] T b4ebd2b043024a548005c8b6b00b0449 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:25.309407 25801 leader_election.cc:288] T 3d89396576884f7c8901dba54f783947 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:25.341315 25789 leader_election.cc:288] T 71f461a6e2ff434993b78a2d077f0abd P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:25.341923 24540 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 1121) took 4274 ms. Trace:
I20260925 03:04:25.344732 24540 rpcz_store.cc:276] 0925 03:04:21.067429 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:21.072243 (+ 4814us) service_pool.cc:227] Handling call
0925 03:04:21.080269 (+ 8026us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:04:23.552841 (+2472572us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:23.603210 (+ 50369us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:04:23.662641 (+ 59431us) log.cc:842] Serialized 26 byte log entry
0925 03:04:24.061201 (+398560us) raft_consensus.cc:1631] Marking committed up to 2
0925 03:04:24.067610 (+ 6409us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:25.341426 (+1273816us) spinlock_profiling.cc:243] Waited 175 ms on lock 0x7b540036bd54. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d0f51 00007f43764cf99e 00007f437b4112e9 00007f4374af7684 00007f4374af7612 00007f4374af7541 00007f4374af74bc 00007f4374af6772 00007f43748fa94c 00007f43748f9d86
0925 03:04:25.341482 (+ 56us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:04:25.341538 (+ 56us) raft_consensus.cc:1671] finished
0925 03:04:25.341551 (+ 13us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:25.341820 (+ 269us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":88479,"raft.run_cpu_time_us":208,"raft.run_wall_time_us":5155,"spinlock_wait_cycles":403641088,"thread_start_us":2610668,"threads_started":2,"wal-append.queue_time_us":553773}
I20260925 03:04:25.253129 25845 raft_consensus.cc:729] T e6b6ec010caf4689b8c90449764f5cf9 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:04:25.297699 25733 leader_election.cc:288] T 21c43af5e6844fdaacc8c625fa01fd8d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:25.358901 25793 leader_election.cc:288] T 420163a6cdce412ebc21c25fe3835638 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:25.360704 24621 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38570 (request call id 1147) took 1253 ms. Trace:
W20260925 03:04:25.363963 25819 consensus_meta.cc:287] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c: Time spent flushing consensus metadata: real 0.599s user 0.004s sys 0.004s
I20260925 03:04:25.343571 25740 leader_election.cc:288] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:25.346113 25666 leader_election.cc:288] T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:25.383451 25790 leader_election.cc:288] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
W20260925 03:04:25.387729 25761 consensus_meta.cc:287] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c: Time spent flushing consensus metadata: real 0.980s user 0.000s sys 0.013s
W20260925 03:04:25.408357 24548 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 1135) took 2274 ms (client timeout 2193 ms). Trace:
I20260925 03:04:25.385946 25845 consensus_queue.cc:260] T e6b6ec010caf4689b8c90449764f5cf9 P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
W20260925 03:04:25.406385 25636 consensus_meta.cc:287] T 0b5df303cd6a4d59aca5c18bbbff24ed P eeab93abfdb141a895002ec9f312296c: Time spent flushing consensus metadata: real 1.025s user 0.012s sys 0.000s
W20260925 03:04:25.440574 24513 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.963s user 0.038s sys 0.078s
I20260925 03:04:25.387085 24621 rpcz_store.cc:276] 0925 03:04:24.117655 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:24.140206 (+ 22551us) service_pool.cc:227] Handling call
0925 03:04:25.360443 (+1220237us) spinlock_profiling.cc:243] Waited 252 ms on lock 0x7b54000a5854. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d42ae 00007f437b411c27 00007f4374af9c04 00007f4374af9b92 00007f4374af9ac1 00007f4374af9a3c 00007f4374af8cf2 00007f43748fa94c 00007f43748f9d86 00007f43748f972f
0925 03:04:25.360601 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":578661248}
I20260925 03:04:25.355096 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1e85248d3b04b979dd080caeeb02fdb" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
W20260925 03:04:25.441169 24513 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.968s user 0.039s sys 0.078s
I20260925 03:04:25.443056 25785 raft_consensus.cc:504] T 04e068a0f80941dc9138cee063f53bab P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
W20260925 03:04:25.457995 25751 consensus_meta.cc:287] T 7993f9c3e2a44f408bbf5d1e395e2306 P eeab93abfdb141a895002ec9f312296c: Time spent flushing consensus metadata: real 0.572s user 0.011s sys 0.000s
I20260925 03:04:25.458680 25751 raft_consensus.cc:504] T 7993f9c3e2a44f408bbf5d1e395e2306 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:25.462132 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e068a0f80941dc9138cee063f53bab" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:25.462818 24617 raft_consensus.cc:3037] T 04e068a0f80941dc9138cee063f53bab P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Advancing to term 2
W20260925 03:04:25.413368 24548 rpcz_store.cc:269] 0925 03:04:23.103917 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:23.525484 (+421567us) service_pool.cc:227] Handling call
0925 03:04:25.378311 (+1852827us) spinlock_profiling.cc:243] Waited 460 ms on lock 0x7b540036c254. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d42ae 00007f437b411c27 00007f4374af9c04 00007f4374af9b92 00007f4374af9ac1 00007f4374af9a3c 00007f4374af8cf2 00007f43748fa94c 00007f43748f9d86 00007f43748f972f
0925 03:04:25.378459 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1057974144}
I20260925 03:04:25.458195 25785 leader_election.cc:288] T 04e068a0f80941dc9138cee063f53bab P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 2 election: Requested vote from peers eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:25.482295 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f68804eb5a244d87bb395d50683c6012" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
W20260925 03:04:25.475196 25648 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.162s user 0.015s sys 0.027s
W20260925 03:04:25.489073 25648 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.186s user 0.016s sys 0.027s
I20260925 03:04:25.483040 24548 raft_consensus.cc:2454] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 2.
I20260925 03:04:25.478940 25793 raft_consensus.cc:482] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:25.496081 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e8af2641e124f9589c5ccfdac785f63" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:25.493517 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
W20260925 03:04:25.496996 24439 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.153s user 0.021s sys 0.027s
W20260925 03:04:25.500763 24439 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.157s user 0.021s sys 0.027s
I20260925 03:04:25.463231 25786 leader_election.cc:288] T 29637f5a4a504ce6ba29033ccd0cd716 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:25.508981 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ac466793d094325ace224708b2158ed" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:25.495644 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "550118bc1a0641ce9803bdbf43f2069f" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:25.496872 25845 raft_consensus.cc:3037] T e6b6ec010caf4689b8c90449764f5cf9 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Advancing to term 3
W20260925 03:04:25.500159 24470 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.323s user 0.050s sys 0.090s
W20260925 03:04:25.510866 24470 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.334s user 0.050s sys 0.090s
W20260925 03:04:25.513115 24514 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.192s user 0.014s sys 0.019s
I20260925 03:04:25.498308 24469 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55560 (request call id 1110) took 1558 ms. Trace:
W20260925 03:04:25.510277 24586 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.745s user 0.011s sys 0.020s
W20260925 03:04:25.542125 25196 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.019s user 0.023s sys 0.032s
I20260925 03:04:25.578218 24546 raft_consensus.cc:1261] T f22ddd7d1c444dc58f45a1fb0b3e02bc P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260925 03:04:25.510337 24540 raft_consensus.cc:1261] T 11bb46bce3564841b7674e6a693bd2e6 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20260925 03:04:25.585783 24617 raft_consensus.cc:2454] T 04e068a0f80941dc9138cee063f53bab P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 2.
W20260925 03:04:25.587035 24514 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.267s user 0.014s sys 0.019s
W20260925 03:04:25.591898 24586 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.827s user 0.011s sys 0.020s
I20260925 03:04:25.514552 24539 raft_consensus.cc:2454] T 550118bc1a0641ce9803bdbf43f2069f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 1.
W20260925 03:04:25.576108 24471 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.504s user 0.007s sys 0.025s
W20260925 03:04:25.613488 24471 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.542s user 0.008s sys 0.026s
I20260925 03:04:25.606634 24540 pending_rounds.cc:76] T 11bb46bce3564841b7674e6a693bd2e6 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Aborting all ops after (but not including) 1
W20260925 03:04:25.594245 25196 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.071s user 0.023s sys 0.032s
I20260925 03:04:25.508131 25793 raft_consensus.cc:504] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } }
I20260925 03:04:25.722155 24540 pending_rounds.cc:97] T 11bb46bce3564841b7674e6a693bd2e6 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260925 03:04:25.722311 24548 raft_consensus.cc:1261] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
W20260925 03:04:25.605479 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 1.36524e+06us
I20260925 03:04:25.577378 25698 raft_consensus.cc:482] T 442d4a29b1d743599f4b6b4871e26668 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:25.591449 24469 rpcz_store.cc:276] 0925 03:04:23.938798 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:25.355830 (+1417032us) service_pool.cc:227] Handling call
0925 03:04:25.445480 (+ 89650us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:25.469562 (+ 24082us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:25.474157 (+ 4595us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:25.475316 (+ 1159us) raft_consensus.cc:1631] Marking committed up to 1
0925 03:04:25.480704 (+ 5388us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:25.491261 (+ 10557us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:25.496845 (+ 5584us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260925 03:04:25.732005 24472 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.875s user 0.000s sys 0.028s
W20260925 03:04:25.735592 24472 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.879s user 0.000s sys 0.028s
W20260925 03:04:25.715063 25195 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.913s user 0.002s sys 0.013s
I20260925 03:04:25.741385 24468 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 571) took 2819 ms. Trace:
I20260925 03:04:25.612653 25751 leader_election.cc:288] T 7993f9c3e2a44f408bbf5d1e395e2306 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
W20260925 03:04:25.747630 25794 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.522s user 0.018s sys 0.016s
I20260925 03:04:25.736516 25738 raft_consensus.cc:2726] T b627ecb4a5384ef392aa695e06599c39 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260925 03:04:25.744105 24468 rpcz_store.cc:276] 0925 03:04:22.923797 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:22.923911 (+ 114us) service_pool.cc:227] Handling call
0925 03:04:22.935172 (+ 11261us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:23.971149 (+1035977us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:25.741097 (+1769948us) spinlock_profiling.cc:243] Waited 625 ms on lock 0x7b54003f6e54. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d0f51 00007f43764cf99e 00007f437b4112e9 00007f4374af7684 00007f4374af7612 00007f4374af7541 00007f4374af74bc 00007f4374af6772 00007f43748fa94c 00007f43748f9d86
0925 03:04:25.741318 (+ 221us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1437353856}
I20260925 03:04:25.750818 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442d4a29b1d743599f4b6b4871e26668" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:25.720863 25642 raft_consensus.cc:482] T 25102ba08e4e49388f6176badcd10a25 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:25.731360 24540 raft_consensus.cc:2866] T 11bb46bce3564841b7674e6a693bd2e6 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260925 03:04:25.732730 25761 raft_consensus.cc:482] T 0b5df303cd6a4d59aca5c18bbbff24ed P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:25.781235 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7993f9c3e2a44f408bbf5d1e395e2306" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:25.733001 25798 raft_consensus.cc:2781] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260925 03:04:25.724045 25797 leader_election.cc:288] T c88cbd39e01f44628a094af516be2833 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
W20260925 03:04:25.744881 25195 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.984s user 0.002s sys 0.013s
I20260925 03:04:25.726794 25162 leader_election.cc:302] T 04e068a0f80941dc9138cee063f53bab P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:25.796173 25163 leader_election.cc:302] T a4abe0ea757e4886be4945c5001e529a P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
W20260925 03:04:25.782568 25685 consensus_meta.cc:287] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5: Time spent flushing consensus metadata: real 0.646s user 0.002s sys 0.008s
I20260925 03:04:25.800786 25163 leader_election.cc:302] T 8ac466793d094325ace224708b2158ed P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
W20260925 03:04:25.783274 25794 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.576s user 0.019s sys 0.017s
I20260925 03:04:25.780350 24466 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55560 (request call id 1116) took 1702 ms. Trace:
I20260925 03:04:25.734740 25698 raft_consensus.cc:504] T 442d4a29b1d743599f4b6b4871e26668 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:04:25.802977 24437 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.934s user 0.000s sys 0.018s
W20260925 03:04:25.820381 24437 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.951s user 0.002s sys 0.019s
W20260925 03:04:25.800251 25799 consensus_meta.cc:287] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent flushing consensus metadata: real 0.684s user 0.000s sys 0.010s
W20260925 03:04:25.804121 25165 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.779s user 0.003s sys 0.011s
I20260925 03:04:25.801645 25786 raft_consensus.cc:2726] T a4abe0ea757e4886be4945c5001e529a P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260925 03:04:25.823144 25165 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.798s user 0.003s sys 0.011s
I20260925 03:04:25.823390 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e068a0f80941dc9138cee063f53bab" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:25.805546 25517 raft_consensus.cc:2726] T 8ac466793d094325ace224708b2158ed P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260925 03:04:25.789413 25761 raft_consensus.cc:504] T 0b5df303cd6a4d59aca5c18bbbff24ed P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:04:25.831084 25165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 806484us
I20260925 03:04:25.795413 25642 raft_consensus.cc:504] T 25102ba08e4e49388f6176badcd10a25 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:25.832643 24468 raft_consensus.cc:2454] T 442d4a29b1d743599f4b6b4871e26668 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 3.
W20260925 03:04:25.834110 24511 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.024s user 0.025s sys 0.011s
W20260925 03:04:25.836230 24511 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.026s user 0.026s sys 0.011s
W20260925 03:04:25.838289 24511 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 1.02853e+06us
W20260925 03:04:25.832655 25804 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.844s user 0.041s sys 0.008s
I20260925 03:04:25.806223 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367277daf8d04778b3fa4ed94d71580d" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
W20260925 03:04:25.824162 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 955451us
W20260925 03:04:25.841708 25198 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.755s user 0.003s sys 0.020s
W20260925 03:04:25.844988 25198 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.758s user 0.003s sys 0.020s
W20260925 03:04:25.833894 25791 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.719s user 0.007s sys 0.016s
I20260925 03:04:25.865404 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74443a0a1dc34d88a2f42e3b722db405" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:25.825187 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35992316cb8d4c028333388eff02ecde" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
W20260925 03:04:25.848146 25818 log.cc:924] Time spent T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 1.462s user 0.017s sys 0.028s
I20260925 03:04:25.825409 25799 raft_consensus.cc:504] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:04:25.873102 24367 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.766s user 0.012s sys 0.012s
I20260925 03:04:25.852545 25199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e92f377a79354a6ebb4eec67303c13e3" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:25.884338 25199 raft_consensus.cc:2454] T e92f377a79354a6ebb4eec67303c13e3 P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 1.
I20260925 03:04:25.877674 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74443a0a1dc34d88a2f42e3b722db405" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:25.806521 24466 rpcz_store.cc:276] 0925 03:04:24.061899 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:25.496005 (+1434106us) service_pool.cc:227] Handling call
0925 03:04:25.591530 (+ 95525us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:25.653379 (+ 61849us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:25.723892 (+ 70513us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:25.726442 (+ 2550us) raft_consensus.cc:1631] Marking committed up to 2
0925 03:04:25.737400 (+ 10958us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:25.743297 (+ 5897us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:25.763721 (+ 20424us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260925 03:04:25.885172 24473 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 587) took 1814 ms (client timeout 1952 ms). Trace:
I20260925 03:04:25.885779 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e92f377a79354a6ebb4eec67303c13e3" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:25.886351 24547 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33068 (request call id 50) took 3740 ms. Trace:
I20260925 03:04:25.886821 24547 rpcz_store.cc:276] 0925 03:04:22.146466 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:22.147020 (+ 554us) service_pool.cc:227] Handling call
0925 03:04:22.926900 (+779880us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:04:23.669500 (+742600us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:23.669555 (+ 55us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:04:23.702212 (+ 32657us) log.cc:842] Serialized 26 byte log entry
0925 03:04:23.941428 (+239216us) raft_consensus.cc:1631] Marking committed up to 4
0925 03:04:23.941472 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:24.110706 (+169234us) spinlock_profiling.cc:243] Waited 628 ms on lock 0x7b5400012cd4. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d0f51 00007f43764cf99e 00007f437b4112e9 00007f4374af7684 00007f4374af7612 00007f4374af7541 00007f4374af74bc 00007f4374af6772 00007f43748fa94c 00007f43748f9d86
0925 03:04:24.110761 (+ 55us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:04:25.753014 (+1642253us) spinlock_profiling.cc:243] Waited 34 ms on lock 0x7b44000232cc. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f43733ecf11 00007f43734baa95 00007f43734bb4b2 00007f4376163511 00007f437615ee1a 00007f4376152bee 00007f4376151f36 00007f4376151b58 00007f437615b551 00007f437615b509 00007f437615b499
0925 03:04:25.886124 (+133110us) raft_consensus.cc:1671] finished
0925 03:04:25.886150 (+ 26us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:25.886272 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":18348,"raft.queue_time_us":87138,"raft.run_cpu_time_us":209,"raft.run_wall_time_us":15482,"spinlock_wait_cycles":1523049088,"thread_start_us":265998,"threads_started":2,"wal-append.queue_time_us":651018}
I20260925 03:04:25.885600 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37a157996e944db2906110d87ecb0c54" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
W20260925 03:04:25.845235 25804 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.861s user 0.041s sys 0.009s
I20260925 03:04:25.878806 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:25.882225 25194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:25.882418 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15861cd28e354598a3c6bfc4337c34bb" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:25.883090 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06f4a7eaf6534e4986c9fd35911038e4" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:25.883740 25698 leader_election.cc:288] T 442d4a29b1d743599f4b6b4871e26668 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:25.884492 25642 leader_election.cc:288] T 25102ba08e4e49388f6176badcd10a25 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
W20260925 03:04:25.885496 24473 rpcz_store.cc:269] 0925 03:04:24.070529 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:24.425490 (+354961us) service_pool.cc:227] Handling call
0925 03:04:25.876611 (+1451121us) spinlock_profiling.cc:243] Waited 5.22 ms on lock 0x7b5800001598. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f43733ecf11 00007f4373570851 00007f4373570702 00007f437c460b09 00007f437b39c730 00007f437b46279b 00007f437b462821 00007f437b4121b0 00007f437b411a94 00007f4374af9c04 00007f4374af9b92
0925 03:04:25.885099 (+ 8488us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":12005632}
I20260925 03:04:25.878510 24467 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 585) took 1836 ms. Trace:
I20260925 03:04:25.877157 25855 raft_consensus.cc:482] T 43366a0a2f9d4344bf4e37ffd290ef8a P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:25.833485 25786 raft_consensus.cc:2781] T 04e068a0f80941dc9138cee063f53bab P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 FOLLOWER]: Leader election won for term 2
W20260925 03:04:25.881950 24367 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.777s user 0.012s sys 0.012s
W20260925 03:04:25.888660 25163 leader_election.cc:334] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389): Timed out: RequestConsensusVote RPC to 127.22.193.2:36389 timed out after 1.521s (SENT)
I20260925 03:04:25.886435 24548 raft_consensus.cc:2454] T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 1.
W20260925 03:04:25.874617 25791 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.761s user 0.009s sys 0.016s
I20260925 03:04:25.904013 25163 leader_election.cc:302] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:25.904182 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
W20260925 03:04:25.867213 24436 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.767s user 0.017s sys 0.009s
W20260925 03:04:25.886718 25847 log.cc:924] Time spent T 367277daf8d04778b3fa4ed94d71580d P 1bc1f2e62ab44af995165bd5116a9d43: Append to log took a long time: real 0.104s user 0.000s sys 0.000s
I20260925 03:04:25.891069 24545 raft_consensus.cc:2454] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 3.
I20260925 03:04:25.911399 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5332a5d329493991dff994e70ea92c" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
W20260925 03:04:25.892309 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 565530us
I20260925 03:04:25.914340 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc2df40db08649c9a75e90000b878559" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:25.880498 25649 raft_consensus.cc:482] T 1414dcfe53f64a78ae413531eb8ec117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:25.901077 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b030d84c670b48d5bff337ccc0c77626" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:25.900839 24467 rpcz_store.cc:276] 0925 03:04:24.042080 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:25.202748 (+1160668us) service_pool.cc:227] Handling call
0925 03:04:25.876763 (+674015us) spinlock_profiling.cc:243] Waited 5.59 ms on lock 0x7b540019f854. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d42ae 00007f437b411c27 00007f4374af9c04 00007f4374af9b92 00007f4374af9ac1 00007f4374af9a3c 00007f4374af8cf2 00007f43748fa94c 00007f43748f9d86 00007f43748f972f
0925 03:04:25.878456 (+ 1693us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":12846976}
I20260925 03:04:25.848591 25793 leader_election.cc:288] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:25.904871 24544 raft_consensus.cc:2454] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 3.
W20260925 03:04:25.905244 24436 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.812s user 0.018s sys 0.010s
W20260925 03:04:25.928578 24436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 835159us
I20260925 03:04:25.926252 25786 raft_consensus.cc:686] T 04e068a0f80941dc9138cee063f53bab P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 LEADER]: Becoming Leader. State: Replica: 1bc1f2e62ab44af995165bd5116a9d43, State: Running, Role: LEADER
W20260925 03:04:25.902580 24438 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.811s user 0.021s sys 0.035s
W20260925 03:04:25.891971 25819 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.508s user 0.006s sys 0.011s
I20260925 03:04:25.933256 24619 raft_consensus.cc:1226] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 2. Current term is 3. Ops: []
I20260925 03:04:25.929035 24466 raft_consensus.cc:2396] T 37a157996e944db2906110d87ecb0c54 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260925 03:04:25.926730 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e068a0f80941dc9138cee063f53bab" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:25.917800 25199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71f461a6e2ff434993b78a2d077f0abd" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
W20260925 03:04:25.930377 24438 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.839s user 0.022s sys 0.036s
W20260925 03:04:25.928007 25678 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.065s user 0.015s sys 0.006s
I20260925 03:04:25.927266 24547 raft_consensus.cc:2454] T b030d84c670b48d5bff337ccc0c77626 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 3.
W20260925 03:04:25.891350 25749 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.816s user 0.006s sys 0.013s
I20260925 03:04:25.915025 24617 raft_consensus.cc:2454] T cc2df40db08649c9a75e90000b878559 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 1.
I20260925 03:04:25.899636 25194 raft_consensus.cc:2454] T 7e49248643df4131a1c42e74244e05c2 P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 3.
I20260925 03:04:25.917874 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d89396576884f7c8901dba54f783947" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:25.938565 24620 raft_consensus.cc:1226] T 04e068a0f80941dc9138cee063f53bab P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 1. Current term is 2. Ops: []
I20260925 03:04:25.902376 24541 raft_consensus.cc:2454] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 2.
I20260925 03:04:25.900113 24514 leader_election.cc:302] T 367277daf8d04778b3fa4ed94d71580d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:25.943863 24614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25102ba08e4e49388f6176badcd10a25" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
W20260925 03:04:25.932917 25819 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.549s user 0.007s sys 0.011s
I20260925 03:04:25.917842 25759 raft_consensus.cc:482] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:25.939500 24467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7993f9c3e2a44f408bbf5d1e395e2306" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
W20260925 03:04:25.997318 24436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 51362.3us
I20260925 03:04:25.997859 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c23337a7cb74184a0e57b7095cb6a11" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
W20260925 03:04:25.947713 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 47686.8us
I20260925 03:04:26.002604 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "550118bc1a0641ce9803bdbf43f2069f" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:26.003221 24619 raft_consensus.cc:2454] T 550118bc1a0641ce9803bdbf43f2069f P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 1.
I20260925 03:04:25.941720 25788 raft_consensus.cc:2726] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260925 03:04:25.938086 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
W20260925 03:04:26.007529 25165 leader_election.cc:334] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187): Timed out: RequestConsensusVote RPC to 127.22.193.1:42187 timed out after 1.695s (SENT)
I20260925 03:04:25.942616 24618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c85f5d1d3f004cf5817bb594fd8e6469" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:26.011137 24514 leader_election.cc:302] T f22ddd7d1c444dc58f45a1fb0b3e02bc P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:04:26.009174 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e5819c15eac4544a57e5ee6c9d4f117" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:26.008208 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5f9b503b0ab43999d582015a367441e" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:26.000140 25759 raft_consensus.cc:504] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:26.020329 24467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b627ecb4a5384ef392aa695e06599c39" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:25.998816 25851 raft_consensus.cc:2726] T 367277daf8d04778b3fa4ed94d71580d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260925 03:04:26.008466 25193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4ebd2b043024a548005c8b6b00b0449" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:25.917507 25855 raft_consensus.cc:504] T 43366a0a2f9d4344bf4e37ffd290ef8a P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:25.933302 25649 raft_consensus.cc:504] T 1414dcfe53f64a78ae413531eb8ec117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:04:26.022009 24464 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.566s user 0.031s sys 0.026s
W20260925 03:04:26.004652 25749 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.933s user 0.006s sys 0.013s
I20260925 03:04:26.023953 25199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15861cd28e354598a3c6bfc4337c34bb" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:26.036356 25199 raft_consensus.cc:2454] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 3.
I20260925 03:04:26.000351 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e92f377a79354a6ebb4eec67303c13e3" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:26.020988 24614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f1fac028a144058b92331c0d9b44ebf" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:26.021270 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21c43af5e6844fdaacc8c625fa01fd8d" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
W20260925 03:04:26.009907 25678 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.157s user 0.015s sys 0.006s
I20260925 03:04:26.026158 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9c33fd100ff44c1a3247319a324a044" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
W20260925 03:04:26.012156 25848 log.cc:924] Time spent T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5: Append to log took a long time: real 0.122s user 0.000s sys 0.001s
W20260925 03:04:26.034090 24464 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.578s user 0.032s sys 0.026s
I20260925 03:04:26.021790 25191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6b6ec010caf4689b8c90449764f5cf9" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
W20260925 03:04:26.038748 25737 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.170s user 0.007s sys 0.025s
I20260925 03:04:26.039783 24614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bb46bce3564841b7674e6a693bd2e6" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
W20260925 03:04:26.026842 25827 thread.cc:636] apply [worker] (thread pool) Time spent creating pthread: real 1.645s user 0.034s sys 0.043s
I20260925 03:04:26.079217 25165 leader_election.cc:302] T b1e85248d3b04b979dd080caeeb02fdb P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:26.081344 24471 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 573) took 3083 ms. Trace:
I20260925 03:04:26.081869 25199 raft_consensus.cc:1226] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 2. Current term is 3. Ops: [2.5-2.5]
W20260925 03:04:26.139115 25165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 59982us
I20260925 03:04:26.020800 25817 raft_consensus.cc:3032] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 LEADER]: Stepping down as leader of term 2
W20260925 03:04:26.084098 24436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 45530.2us
I20260925 03:04:26.084756 24619 raft_consensus.cc:2454] T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 3.
I20260925 03:04:26.149927 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f22ddd7d1c444dc58f45a1fb0b3e02bc" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:26.033707 25761 leader_election.cc:288] T 0b5df303cd6a4d59aca5c18bbbff24ed P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:26.029019 25804 raft_consensus.cc:2781] T f22ddd7d1c444dc58f45a1fb0b3e02bc P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260925 03:04:26.087045 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9c33fd100ff44c1a3247319a324a044" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:26.108266 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60d45b7d467d4e1ab92367f6d7d52b15" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
W20260925 03:04:26.026711 25782 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.564s user 0.015s sys 0.011s
W20260925 03:04:26.167096 25782 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.704s user 0.015s sys 0.011s
I20260925 03:04:26.167780 24470 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55560 (request call id 1105) took 2319 ms. Trace:
I20260925 03:04:26.164366 25194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4ebd2b043024a548005c8b6b00b0449" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
W20260925 03:04:26.152199 25827 thread.cc:602] apply [worker] (thread pool) Time spent starting thread: real 1.859s user 0.035s sys 0.043s
I20260925 03:04:26.169116 25194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e54ca1757ee4e42956a9cebbeca4f62" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:26.169973 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f1fac028a144058b92331c0d9b44ebf" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:26.170559 25759 leader_election.cc:288] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:26.170855 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e5819c15eac4544a57e5ee6c9d4f117" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:26.171587 24547 raft_consensus.cc:2454] T 8e5819c15eac4544a57e5ee6c9d4f117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 1.
I20260925 03:04:26.172534 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f22ddd7d1c444dc58f45a1fb0b3e02bc" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:26.172956 24472 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55560 (request call id 1106) took 2324 ms. Trace:
I20260925 03:04:26.174050 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b8e0d260a884ecdb20f79de46c36b19" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:26.156816 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ac285175464fbda67211d9574db415" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
W20260925 03:04:26.124974 25737 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.278s user 0.007s sys 0.025s
I20260925 03:04:26.152601 25785 raft_consensus.cc:2726] T b1e85248d3b04b979dd080caeeb02fdb P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 03:04:26.175347 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e54ca1757ee4e42956a9cebbeca4f62" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:26.158059 24614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c81a1b3b616492ab6c964f45af36077" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:26.176071 24621 raft_consensus.cc:2454] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 3.
I20260925 03:04:26.078831 25700 raft_consensus.cc:482] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:26.176980 25700 raft_consensus.cc:504] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:26.177732 25199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c81a1b3b616492ab6c964f45af36077" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:26.179101 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:26.179481 25700 leader_election.cc:288] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:26.166352 25648 raft_consensus.cc:482] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:26.180938 25648 raft_consensus.cc:504] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:26.181264 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35992316cb8d4c028333388eff02ecde" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:26.173316 24472 rpcz_store.cc:276] 0925 03:04:23.848665 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:23.848902 (+ 237us) service_pool.cc:227] Handling call
0925 03:04:23.930082 (+ 81180us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:04:23.994164 (+ 64082us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:23.994199 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:04:24.011335 (+ 17136us) log.cc:842] Serialized 26 byte log entry
0925 03:04:24.856374 (+845039us) spinlock_profiling.cc:243] Waited 6.58 ms on lock 0x7b1800374500. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f437c4478b1 00007f437c454661 00007f437358a345 00007f437c3c1a39 00007f43735a78bb 00007f43735a5cbf 00007f43735a7e22 00007f43761515bb 00007f4376157063 00007f4376157411 00007f43764a45cf
0925 03:04:25.736050 (+879676us) raft_consensus.cc:1631] Marking committed up to 1
0925 03:04:25.736094 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:25.736184 (+ 90us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:04:26.172613 (+436429us) raft_consensus.cc:1671] finished
0925 03:04:26.172634 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:26.172900 (+ 266us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":85858,"raft.run_cpu_time_us":185,"raft.run_wall_time_us":12202,"spinlock_wait_cycles":15125888,"thread_start_us":879454,"threads_started":1,"wal-append.queue_time_us":1986559}
I20260925 03:04:26.168493 25191 raft_consensus.cc:2379] T e6b6ec010caf4689b8c90449764f5cf9 P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 3: Already voted for candidate 1bc1f2e62ab44af995165bd5116a9d43 in this term.
I20260925 03:04:26.158761 24464 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55560 (request call id 1108) took 2253 ms. Trace:
I20260925 03:04:26.157464 24548 raft_consensus.cc:1261] T b030d84c670b48d5bff337ccc0c77626 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260925 03:04:26.163990 25190 raft_consensus.cc:2932] T 7e49248643df4131a1c42e74244e05c2 P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:26.183867 24614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5332a5d329493991dff994e70ea92c" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:26.180224 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c88cbd39e01f44628a094af516be2833" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:26.184572 24544 raft_consensus.cc:2454] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 3.
I20260925 03:04:26.184679 24464 rpcz_store.cc:276] 0925 03:04:23.905007 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:25.316358 (+1411351us) service_pool.cc:227] Handling call
0925 03:04:25.448964 (+132606us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:25.449168 (+ 204us) raft_consensus.cc:1520] Early marking committed up to index 2
0925 03:04:25.455720 (+ 6552us) log.cc:842] Serialized 14 byte log entry
0925 03:04:26.158434 (+702714us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:26.158485 (+ 51us) raft_consensus.cc:1631] Marking committed up to 2
0925 03:04:26.158528 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:26.158593 (+ 65us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:26.158703 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":73948,"prepare.queue_time_us":9131,"prepare.run_cpu_time_us":77,"prepare.run_wall_time_us":79,"raft.queue_time_us":74928,"raft.run_cpu_time_us":110,"raft.run_wall_time_us":111,"thread_start_us":628343,"threads_started":1}
I20260925 03:04:26.137494 24471 rpcz_store.cc:276] 0925 03:04:22.998246 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:22.999216 (+ 970us) service_pool.cc:227] Handling call
0925 03:04:23.013172 (+ 13956us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:23.881319 (+868147us) raft_consensus.cc:1520] Early marking committed up to index 5
0925 03:04:24.071434 (+190115us) log.cc:842] Serialized 14 byte log entry
0925 03:04:25.724570 (+1653136us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:25.724624 (+ 54us) raft_consensus.cc:1631] Marking committed up to 5
0925 03:04:25.724667 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:26.081119 (+356452us) spinlock_profiling.cc:243] Waited 868 ms on lock 0x7b54002c0954. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d0f51 00007f43764cf99e 00007f437b4112e9 00007f4374af7684 00007f4374af7612 00007f4374af7541 00007f4374af74bc 00007f4374af6772 00007f43748fa94c 00007f43748f9d86
0925 03:04:26.081163 (+ 44us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:26.081287 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9698,"prepare.queue_time_us":589488,"prepare.run_cpu_time_us":60,"prepare.run_wall_time_us":4459,"raft.queue_time_us":9836,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":3763,"spinlock_wait_cycles":1996079488,"thread_start_us":1678385,"threads_started":2,"wal-append.queue_time_us":1769524}
I20260925 03:04:26.080940 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be4e0aa984f4f2b81a58e382e434ad5" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:26.080792 25678 raft_consensus.cc:482] T a7aa47c48404465b94a36fd819fe3c7a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:26.072580 25786 consensus_queue.cc:237] T 04e068a0f80941dc9138cee063f53bab P 1bc1f2e62ab44af995165bd5116a9d43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:26.285082 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e8af2641e124f9589c5ccfdac785f63" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:26.367729 24614 raft_consensus.cc:2454] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 3.
W20260925 03:04:26.256861 24546 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.534s user 0.029s sys 0.047s
W20260925 03:04:26.370101 24546 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.647s user 0.032s sys 0.053s
W20260925 03:04:26.369500 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 493001us
I20260925 03:04:26.371474 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4abe0ea757e4886be4945c5001e529a" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:26.205720 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc2df40db08649c9a75e90000b878559" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:26.372988 24545 raft_consensus.cc:2454] T cc2df40db08649c9a75e90000b878559 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 1.
I20260925 03:04:26.373912 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abd5b8dc1b7e49e3aa9af900d73ff0e7" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:26.374963 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06f4a7eaf6534e4986c9fd35911038e4" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:26.375490 24545 raft_consensus.cc:3037] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:26.257932 24436 leader_election.cc:302] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5; no voters:
I20260925 03:04:26.160432 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35992316cb8d4c028333388eff02ecde" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:26.377980 24619 raft_consensus.cc:2454] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 3.
W20260925 03:04:26.382479 24436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.4:43979, user_credentials={real_user=slave}} blocked reactor thread for 124862us
I20260925 03:04:26.389663 24614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e725b82985746b39c0cc8f671c95353" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
W20260925 03:04:26.349767 25845 consensus_meta.cc:287] T e6b6ec010caf4689b8c90449764f5cf9 P 869b0a8575b54d08ae2817fad5dd42d5: Time spent flushing consensus metadata: real 0.830s user 0.008s sys 0.003s
I20260925 03:04:26.168193 24470 rpcz_store.cc:276] 0925 03:04:23.848379 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:23.848514 (+ 135us) service_pool.cc:227] Handling call
0925 03:04:23.878218 (+ 29704us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:04:24.088003 (+209785us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:24.088038 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:04:24.105225 (+ 17187us) log.cc:842] Serialized 26 byte log entry
0925 03:04:25.511388 (+1406163us) raft_consensus.cc:1631] Marking committed up to 1
0925 03:04:25.511429 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:25.511497 (+ 68us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:04:26.167571 (+656074us) raft_consensus.cc:1671] finished
0925 03:04:26.167592 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:26.167726 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":19182,"raft.queue_time_us":119163,"raft.run_cpu_time_us":173,"raft.run_wall_time_us":6786,"thread_start_us":1509560,"threads_started":2,"wal-append.queue_time_us":1721512}
I20260925 03:04:26.393620 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "293ef69caf514ac98510a68495f54eda" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:26.396970 24548 raft_consensus.cc:2454] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 2.
W20260925 03:04:26.158157 25853 log.cc:924] Time spent T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5: Append to log took a long time: real 0.280s user 0.000s sys 0.001s
I20260925 03:04:26.400709 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4ebd2b043024a548005c8b6b00b0449" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:26.402122 24614 raft_consensus.cc:2396] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260925 03:04:26.161826 25804 raft_consensus.cc:482] T f22ddd7d1c444dc58f45a1fb0b3e02bc P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:26.162395 25817 raft_consensus.cc:729] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:04:26.175369 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5332a5d329493991dff994e70ea92c" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:26.403254 24473 raft_consensus.cc:2454] T c88cbd39e01f44628a094af516be2833 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 3.
I20260925 03:04:26.329380 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25102ba08e4e49388f6176badcd10a25" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:26.258731 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:26.358404 25641 leader_election.cc:288] T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:26.185997 25197 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48408 (request call id 110) took 2087 ms. Trace:
I20260925 03:04:26.295898 25648 leader_election.cc:288] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:26.269526 25799 leader_election.cc:288] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:26.284202 24618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7691c6b475428c82d6181d693d26f3" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:26.205314 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48ca6ccd98194cf2af7b5278d15bf4ea" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:26.340613 25196 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49794 (request call id 123) took 2063 ms. Trace:
I20260925 03:04:26.412248 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceab6b0fccd54e909a64f66fe81a9a1f" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:26.393649 25844 raft_consensus.cc:2781] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260925 03:04:26.301921 25162 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 679558us
I20260925 03:04:26.412264 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b5df303cd6a4d59aca5c18bbbff24ed" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:26.440095 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e725b82985746b39c0cc8f671c95353" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:26.488355 24619 raft_consensus.cc:2396] T 0b5df303cd6a4d59aca5c18bbbff24ed P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260925 03:04:26.394526 25678 raft_consensus.cc:504] T a7aa47c48404465b94a36fd819fe3c7a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:26.392652 25794 raft_consensus.cc:482] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:26.447192 24511 leader_election.cc:302] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 1bc1f2e62ab44af995165bd5116a9d43, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:26.406730 25741 raft_consensus.cc:482] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:26.451997 24473 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 600) took 2177 ms. Trace:
I20260925 03:04:26.399593 25854 raft_consensus.cc:482] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:26.452306 24470 raft_consensus.cc:1226] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260925 03:04:26.403010 25195 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49794 (request call id 121) took 2141 ms. Trace:
I20260925 03:04:26.410523 25804 raft_consensus.cc:3037] T f22ddd7d1c444dc58f45a1fb0b3e02bc P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:26.489948 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5278a60f97da4e10aecd50a8d9a9a43c" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:26.445256 24614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be4e0aa984f4f2b81a58e382e434ad5" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:26.413857 25845 raft_consensus.cc:482] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:26.488849 24548 raft_consensus.cc:2454] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 3.
I20260925 03:04:26.492564 24546 raft_consensus.cc:2379] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c in current term 3: Already voted for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in this term.
I20260925 03:04:26.413235 25791 raft_consensus.cc:482] T c88cbd39e01f44628a094af516be2833 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:26.493665 24473 rpcz_store.cc:276] 0925 03:04:24.275085 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:26.414402 (+2139317us) service_pool.cc:227] Handling call
0925 03:04:26.451525 (+ 37123us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:26.451676 (+ 151us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:26.451702 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:26.451736 (+ 34us) raft_consensus.cc:1631] Marking committed up to 2
0925 03:04:26.451765 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:26.451814 (+ 49us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:26.451917 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260925 03:04:26.499408 24511 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.4:43979, user_credentials={real_user=slave}} blocked reactor thread for 52310.1us
W20260925 03:04:26.501092 25198 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.612s user 0.006s sys 0.029s
I20260925 03:04:26.501586 24470 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 599) took 2250 ms. Trace:
I20260925 03:04:26.598582 24470 rpcz_store.cc:276] 0925 03:04:24.252430 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:26.396491 (+2144061us) service_pool.cc:227] Handling call
0925 03:04:26.452093 (+ 55602us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:04:26.501288 (+ 49195us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:26.501522 (+ 234us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260925 03:04:26.599442 25198 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.710s user 0.006s sys 0.029s
I20260925 03:04:26.599689 24614 raft_consensus.cc:2454] T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 1.
W20260925 03:04:26.597608 24514 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.518s user 0.010s sys 0.020s
I20260925 03:04:26.600292 24619 raft_consensus.cc:2454] T 5278a60f97da4e10aecd50a8d9a9a43c P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 1.
W20260925 03:04:26.600714 24514 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.535s user 0.010s sys 0.020s
W20260925 03:04:26.607252 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 542710us
I20260925 03:04:26.610172 25163 leader_election.cc:302] T 7e49248643df4131a1c42e74244e05c2 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:04:26.599200 24511 leader_election.cc:302] T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:04:26.602303 24473 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38920 (request call id 48) took 1869 ms. Trace:
I20260925 03:04:26.613310 24473 rpcz_store.cc:276] 0925 03:04:24.733041 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:26.601494 (+1868453us) service_pool.cc:227] Handling call
0925 03:04:26.601845 (+ 351us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:26.602015 (+ 170us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:26.602039 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:26.602079 (+ 40us) raft_consensus.cc:1631] Marking committed up to 5
0925 03:04:26.602107 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:26.602154 (+ 47us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:26.602247 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:26.600359 25198 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49794 (request call id 124) took 1628 ms. Trace:
I20260925 03:04:26.490478 25844 raft_consensus.cc:482] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:26.602499 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "420163a6cdce412ebc21c25fe3835638" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:26.605563 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06f4a7eaf6534e4986c9fd35911038e4" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:26.488440 25855 leader_election.cc:288] T 43366a0a2f9d4344bf4e37ffd290ef8a P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:26.637494 24619 raft_consensus.cc:2454] T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 3.
I20260925 03:04:26.603006 24614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21c43af5e6844fdaacc8c625fa01fd8d" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:26.614650 25198 rpcz_store.cc:276] 0925 03:04:24.972138 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:24.972381 (+ 243us) service_pool.cc:227] Handling call
0925 03:04:24.972959 (+ 578us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:24.973189 (+ 230us) raft_consensus.cc:1520] Early marking committed up to index 4
0925 03:04:25.084548 (+111359us) log.cc:842] Serialized 14 byte log entry
0925 03:04:26.599877 (+1515329us) log.cc:842] Serialized 14 byte log entry
0925 03:04:26.600046 (+ 169us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:26.600085 (+ 39us) raft_consensus.cc:1631] Marking committed up to 4
0925 03:04:26.600126 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:26.600189 (+ 63us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:26.600305 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":18653,"prepare.queue_time_us":46372,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":5163,"raft.queue_time_us":661,"raft.run_cpu_time_us":165,"raft.run_wall_time_us":18398,"thread_start_us":1468767,"threads_started":2,"wal-append.queue_time_us":937518}
I20260925 03:04:26.637372 24544 raft_consensus.cc:2454] T 420163a6cdce412ebc21c25fe3835638 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 3.
I20260925 03:04:26.640184 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15861cd28e354598a3c6bfc4337c34bb" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:26.641091 24614 raft_consensus.cc:2454] T 21c43af5e6844fdaacc8c625fa01fd8d P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 1.
I20260925 03:04:26.643720 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5f9b503b0ab43999d582015a367441e" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:26.648373 24473 raft_consensus.cc:1261] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Refusing update from remote peer 1bc1f2e62ab44af995165bd5116a9d43: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260925 03:04:26.649080 24473 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38920 (request call id 50) took 1915 ms. Trace:
W20260925 03:04:26.649309 24617 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.571s user 0.032s sys 0.047s
W20260925 03:04:26.649758 24617 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.571s user 0.032s sys 0.047s
I20260925 03:04:26.649950 24548 raft_consensus.cc:2362] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 3. Re-sending same reply.
I20260925 03:04:26.649408 24473 rpcz_store.cc:276] 0925 03:04:24.733562 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:26.614293 (+1880731us) service_pool.cc:227] Handling call
0925 03:04:26.648031 (+ 33738us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:26.648829 (+ 798us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:26.649024 (+ 195us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260925 03:04:26.491494 25874 log.cc:924] Time spent T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5: Append to log took a long time: real 0.199s user 0.000s sys 0.001s
I20260925 03:04:26.656878 24470 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38920 (request call id 47) took 1924 ms. Trace:
W20260925 03:04:26.597368 24587 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.512s user 0.017s sys 0.027s
I20260925 03:04:26.528990 25846 raft_consensus.cc:2726] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260925 03:04:26.488270 24618 raft_consensus.cc:2454] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 3.
I20260925 03:04:26.660249 24470 rpcz_store.cc:276] 0925 03:04:24.732728 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:26.603627 (+1870899us) service_pool.cc:227] Handling call
0925 03:04:26.656378 (+ 52751us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:26.656555 (+ 177us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:26.656581 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:26.656618 (+ 37us) raft_consensus.cc:1631] Marking committed up to 5
0925 03:04:26.656645 (+ 27us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:26.656721 (+ 76us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:26.656821 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:26.502827 24540 raft_consensus.cc:2454] T 48ca6ccd98194cf2af7b5278d15bf4ea P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 1.
W20260925 03:04:26.645313 25685 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.579s user 0.021s sys 0.054s
W20260925 03:04:26.530889 25866 log.cc:924] Time spent T 29637f5a4a504ce6ba29033ccd0cd716 P 1bc1f2e62ab44af995165bd5116a9d43: Append to log took a long time: real 0.450s user 0.001s sys 0.000s
I20260925 03:04:26.640758 25844 raft_consensus.cc:3037] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:26.695761 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e92f377a79354a6ebb4eec67303c13e3" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:26.641644 25845 raft_consensus.cc:504] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:26.645416 25851 raft_consensus.cc:2781] T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 03:04:26.697150 25851 raft_consensus.cc:482] T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:26.697302 24469 raft_consensus.cc:2454] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 3.
I20260925 03:04:26.697557 25851 raft_consensus.cc:3037] T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:26.531204 25649 leader_election.cc:288] T 1414dcfe53f64a78ae413531eb8ec117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
W20260925 03:04:26.665241 24587 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.656s user 0.017s sys 0.028s
I20260925 03:04:26.638051 25791 raft_consensus.cc:504] T c88cbd39e01f44628a094af516be2833 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:04:26.699445 25859 log.cc:924] Time spent T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.245s user 0.000s sys 0.000s
W20260925 03:04:26.700490 25879 log.cc:924] Time spent T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5: Append to log took a long time: real 0.285s user 0.000s sys 0.000s
I20260925 03:04:26.499338 25794 raft_consensus.cc:504] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
W20260925 03:04:26.701252 25831 log.cc:924] Time spent T 74443a0a1dc34d88a2f42e3b722db405 P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 2.321s user 0.005s sys 0.013s
I20260925 03:04:26.702061 24470 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 609) took 1091 ms. Trace:
I20260925 03:04:26.702428 24470 rpcz_store.cc:276] 0925 03:04:25.610191 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:26.677073 (+1066882us) service_pool.cc:227] Handling call
0925 03:04:26.701595 (+ 24522us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:26.701760 (+ 165us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:26.701784 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:26.701817 (+ 33us) raft_consensus.cc:1631] Marking committed up to 1
0925 03:04:26.701843 (+ 26us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:26.701890 (+ 47us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:26.702008 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260925 03:04:26.705921 25876 log.cc:924] Time spent T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43: Append to log took a long time: real 0.205s user 0.001s sys 0.000s
W20260925 03:04:26.706987 25875 log.cc:924] Time spent T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.066s user 0.000s sys 0.000s
I20260925 03:04:26.660897 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "100c113a9fd84cc8adf2ec7b33ea0093" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
W20260925 03:04:26.671852 25763 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 2.291s user 0.056s sys 0.083s
I20260925 03:04:26.661582 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74443a0a1dc34d88a2f42e3b722db405" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:26.718492 24545 raft_consensus.cc:2396] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260925 03:04:26.664400 25648 raft_consensus.cc:482] T edcf6e6a422d4d198d4fbb0f7d3d845d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
W20260925 03:04:26.668550 24465 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1119) took 2561 ms (client timeout 2086 ms). Trace:
W20260925 03:04:26.676862 25737 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.502s user 0.007s sys 0.032s
W20260925 03:04:26.787568 25737 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.612s user 0.007s sys 0.032s
I20260925 03:04:26.487289 25196 rpcz_store.cc:276] 0925 03:04:24.277041 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:24.297216 (+ 20175us) service_pool.cc:227] Handling call
0925 03:04:24.407147 (+109931us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:24.474431 (+ 67284us) raft_consensus.cc:1520] Early marking committed up to index 5
0925 03:04:25.739130 (+1264699us) log.cc:842] Serialized 14 byte log entry
0925 03:04:26.269154 (+530024us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:26.279492 (+ 10338us) raft_consensus.cc:1631] Marking committed up to 5
0925 03:04:26.292387 (+ 12895us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:26.302796 (+ 10409us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:26.340020 (+ 37224us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":1362959,"prepare.run_cpu_time_us":3982,"prepare.run_wall_time_us":6368,"raft.queue_time_us":16439,"raft.run_cpu_time_us":222,"raft.run_wall_time_us":13217,"thread_start_us":1646324,"threads_started":2,"wal-append.queue_time_us":704837}
W20260925 03:04:26.745271 24542 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 1142) took 3080 ms (client timeout 2258 ms). Trace:
W20260925 03:04:26.694308 25685 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.646s user 0.021s sys 0.055s
I20260925 03:04:26.712263 24544 raft_consensus.cc:2932] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:26.690714 25809 raft_consensus.cc:482] T 43366a0a2f9d4344bf4e37ffd290ef8a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:26.448453 25197 rpcz_store.cc:276] 0925 03:04:24.074591 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:24.085925 (+ 11334us) service_pool.cc:227] Handling call
0925 03:04:24.126027 (+ 40102us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:04:24.145801 (+ 19774us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:24.145841 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:04:24.242212 (+ 96371us) log.cc:842] Serialized 26 byte log entry
0925 03:04:25.314344 (+1072132us) raft_consensus.cc:1631] Marking committed up to 2
0925 03:04:25.314387 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:25.314475 (+ 88us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:04:26.039782 (+725307us) raft_consensus.cc:1671] finished
0925 03:04:26.039818 (+ 36us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:26.162258 (+122440us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":17634,"raft.queue_time_us":5204,"raft.run_cpu_time_us":210,"raft.run_wall_time_us":14303,"thread_start_us":1020049,"threads_started":1,"wal-append.queue_time_us":1438274}
I20260925 03:04:26.636171 25817 consensus_queue.cc:260] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:26.784657 24616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c23337a7cb74184a0e57b7095cb6a11" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:26.785647 24621 raft_consensus.cc:2454] T 74443a0a1dc34d88a2f42e3b722db405 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 3.
W20260925 03:04:26.704681 25869 log.cc:924] Time spent T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.368s user 0.001s sys 0.000s
I20260925 03:04:26.530117 25741 raft_consensus.cc:504] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:26.763317 24588 leader_election.cc:302] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:26.738817 25192 raft_consensus.cc:1226] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 LEADER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 2. Current term is 3. Ops: []
W20260925 03:04:26.961787 24471 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.554s user 0.043s sys 0.105s
W20260925 03:04:26.768217 25878 log.cc:924] Time spent T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43: Append to log took a long time: real 0.107s user 0.000s sys 0.000s
W20260925 03:04:26.796340 24542 rpcz_store.cc:269] 0925 03:04:23.630780 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:23.836951 (+206171us) service_pool.cc:227] Handling call
0925 03:04:26.710877 (+2873926us) spinlock_profiling.cc:243] Waited 300 ms on lock 0x7b54001d3154. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d42ae 00007f437b411c27 00007f4374af9c04 00007f4374af9b92 00007f4374af9ac1 00007f4374af9a3c 00007f4374af8cf2 00007f43748fa94c 00007f43748f9d86 00007f43748f972f
0925 03:04:26.710991 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":690582784}
W20260925 03:04:26.719574 25763 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 2.339s user 0.056s sys 0.084s
I20260925 03:04:26.757921 24541 raft_consensus.cc:2454] T 100c113a9fd84cc8adf2ec7b33ea0093 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 1.
I20260925 03:04:26.531383 25854 raft_consensus.cc:504] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:26.776597 25678 leader_election.cc:288] T a7aa47c48404465b94a36fd819fe3c7a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:26.775979 25845 leader_election.cc:288] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:26.779018 24618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "420163a6cdce412ebc21c25fe3835638" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
W20260925 03:04:26.962220 24471 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.555s user 0.043s sys 0.105s
W20260925 03:04:26.960341 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 197086us
W20260925 03:04:26.787326 24465 rpcz_store.cc:269] 0925 03:04:24.095469 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:24.520567 (+425098us) service_pool.cc:227] Handling call
0925 03:04:26.640304 (+2119737us) spinlock_profiling.cc:243] Waited 210 ms on lock 0x7b540019fd54. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d42ae 00007f437b411c27 00007f4374af9c04 00007f4374af9b92 00007f4374af9ac1 00007f4374af9a3c 00007f4374af8cf2 00007f43748fa94c 00007f43748f9d86 00007f43748f972f
0925 03:04:26.647490 (+ 7186us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":482137984}
W20260925 03:04:27.042286 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 50961.2us
W20260925 03:04:26.967365 25818 log.cc:924] Time spent T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.118s user 0.000s sys 0.000s
I20260925 03:04:26.783918 25844 raft_consensus.cc:504] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:27.047279 24588 leader_election.cc:302] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:27.038604 25192 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48408 (request call id 111) took 2956 ms. Trace:
I20260925 03:04:26.961819 25737 raft_consensus.cc:482] T 3d89396576884f7c8901dba54f783947 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:27.054021 24588 leader_election.cc:302] T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:26.759447 25648 raft_consensus.cc:504] T edcf6e6a422d4d198d4fbb0f7d3d845d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:26.850358 25809 raft_consensus.cc:504] T 43366a0a2f9d4344bf4e37ffd290ef8a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:04:27.099133 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 45197.3us
W20260925 03:04:27.072584 25828 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.671s user 0.011s sys 0.023s
W20260925 03:04:27.101311 25828 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.705s user 0.011s sys 0.023s
I20260925 03:04:27.108181 25648 leader_election.cc:288] T edcf6e6a422d4d198d4fbb0f7d3d845d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:26.959406 25799 consensus_queue.cc:1046] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.131s
I20260925 03:04:26.962805 24616 raft_consensus.cc:2454] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 3.
W20260925 03:04:27.109690 24464 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.702s user 0.039s sys 0.099s
I20260925 03:04:27.097613 24472 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 591) took 2989 ms. Trace:
W20260925 03:04:27.115583 24464 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.708s user 0.040s sys 0.099s
I20260925 03:04:26.964979 25885 raft_consensus.cc:2781] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260925 03:04:27.073367 24437 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.666s user 0.008s sys 0.006s
W20260925 03:04:27.117889 24437 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.711s user 0.013s sys 0.006s
W20260925 03:04:27.116657 24438 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.669s user 0.006s sys 0.019s
W20260925 03:04:27.112337 25869 log.cc:924] Time spent T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.060s user 0.000s sys 0.000s
I20260925 03:04:27.094376 25637 raft_consensus.cc:2781] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader election won for term 3
W20260925 03:04:27.118543 24438 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.694s user 0.007s sys 0.019s
I20260925 03:04:27.118696 24617 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38570 (request call id 1155) took 1115 ms. Trace:
I20260925 03:04:27.123554 24617 rpcz_store.cc:276] 0925 03:04:26.003243 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:26.004196 (+ 953us) service_pool.cc:227] Handling call
0925 03:04:26.004497 (+ 301us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:04:26.004729 (+ 232us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:26.004753 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:04:26.005141 (+ 388us) log.cc:842] Serialized 26 byte log entry
0925 03:04:26.650212 (+645071us) raft_consensus.cc:1631] Marking committed up to 1
0925 03:04:26.650247 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:26.650308 (+ 61us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:04:27.118484 (+468176us) raft_consensus.cc:1671] finished
0925 03:04:27.118500 (+ 16us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:27.118634 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":571525,"threads_started":1,"wal-append.queue_time_us":1015544}
I20260925 03:04:27.125222 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f1fac028a144058b92331c0d9b44ebf" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:26.502239 25195 rpcz_store.cc:276] 0925 03:04:24.261768 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:24.264706 (+ 2938us) service_pool.cc:227] Handling call
0925 03:04:24.359734 (+ 95028us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:24.414695 (+ 54961us) raft_consensus.cc:1520] Early marking committed up to index 5
0925 03:04:25.801253 (+1386558us) log.cc:842] Serialized 14 byte log entry
0925 03:04:26.302880 (+501627us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:26.358613 (+ 55733us) raft_consensus.cc:1631] Marking committed up to 5
0925 03:04:26.374860 (+ 16247us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:26.387938 (+ 13078us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:26.399420 (+ 11482us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":34031,"prepare.queue_time_us":1402936,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":48,"raft.queue_time_us":1231,"raft.run_cpu_time_us":140,"raft.run_wall_time_us":7875,"thread_start_us":1519986,"threads_started":2,"wal-append.queue_time_us":727288}
I20260925 03:04:27.098687 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceab6b0fccd54e909a64f66fe81a9a1f" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:27.128767 24467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1414dcfe53f64a78ae413531eb8ec117" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:27.129755 24467 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 604) took 1520 ms. Trace:
I20260925 03:04:27.130023 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1414dcfe53f64a78ae413531eb8ec117" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:27.130158 24467 rpcz_store.cc:276] 0925 03:04:25.608979 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:26.028223 (+419244us) service_pool.cc:227] Handling call
0925 03:04:27.128553 (+1100330us) spinlock_profiling.cc:243] Waited 17.4 ms on lock 0x7b5800001598. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f43733ecf11 00007f4373570851 00007f4373570702 00007f437c460b09 00007f437b39c730 00007f437b46279b 00007f437b462821 00007f437b4121b0 00007f437b411a94 00007f4374af9c04 00007f4374af9b92
0925 03:04:27.129693 (+ 1140us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":40104832}
I20260925 03:04:27.118732 24618 raft_consensus.cc:2396] T 420163a6cdce412ebc21c25fe3835638 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260925 03:04:27.131048 24617 raft_consensus.cc:2454] T 1414dcfe53f64a78ae413531eb8ec117 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 1.
I20260925 03:04:27.136233 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:27.109536 25761 raft_consensus.cc:2781] T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260925 03:04:27.098487 25737 raft_consensus.cc:504] T 3d89396576884f7c8901dba54f783947 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:27.120560 24616 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38564 (request call id 1136) took 1193 ms. Trace:
I20260925 03:04:27.144897 24616 rpcz_store.cc:276] 0925 03:04:25.926499 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:25.929928 (+ 3429us) service_pool.cc:227] Handling call
0925 03:04:26.784512 (+854584us) spinlock_profiling.cc:243] Waited 1.35 ms on lock 0x7b58001d3d98. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f43733ecf11 00007f4373570851 00007f4373570bce 000000000034aea9 00007f437b39ca30 00007f437b4627b4 00007f437b462821 00007f437b4121b0 00007f437b411a94 00007f4374af9c04 00007f4374af9b92
0925 03:04:27.119535 (+335023us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3110144}
I20260925 03:04:27.145690 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "100c113a9fd84cc8adf2ec7b33ea0093" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:27.146320 24622 raft_consensus.cc:2454] T 100c113a9fd84cc8adf2ec7b33ea0093 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 1.
I20260925 03:04:27.146768 24616 raft_consensus.cc:1261] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260925 03:04:27.147454 24616 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38570 (request call id 1162) took 1122 ms. Trace:
I20260925 03:04:27.146729 24622 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38564 (request call id 1135) took 1230 ms. Trace:
W20260925 03:04:27.147660 24468 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 612) took 1536 ms (client timeout 1616 ms). Trace:
I20260925 03:04:27.138972 24621 raft_consensus.cc:3037] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Advancing to term 3
W20260925 03:04:27.150092 24468 rpcz_store.cc:269] 0925 03:04:25.610896 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:25.750557 (+139661us) service_pool.cc:227] Handling call
0925 03:04:27.147503 (+1396946us) spinlock_profiling.cc:243] Waited 81.1 ms on lock 0x7b54001aaed4. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d42ae 00007f437b411c27 00007f4374af9c04 00007f4374af9b92 00007f4374af9ac1 00007f4374af9a3c 00007f4374af8cf2 00007f43748fa94c 00007f43748f9d86 00007f43748f972f
0925 03:04:27.147603 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":186493184}
I20260925 03:04:27.148027 24622 rpcz_store.cc:276] 0925 03:04:25.916803 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:25.916922 (+ 119us) service_pool.cc:227] Handling call
0925 03:04:27.145530 (+1228608us) spinlock_profiling.cc:243] Waited 1.35 ms on lock 0x7b58001d3d98. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f43733ecf11 00007f4373570851 00007f4373570702 00007f437c460b09 00007f437b39c730 00007f437b46279b 00007f437b462821 00007f437b4121b0 00007f437b411a94 00007f4374af9c04 00007f4374af9b92
0925 03:04:27.146681 (+ 1151us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3099008}
W20260925 03:04:27.145529 25891 log.cc:924] Time spent T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.072s user 0.000s sys 0.000s
I20260925 03:04:27.072724 25192 rpcz_store.cc:276] 0925 03:04:24.081955 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:24.093047 (+ 11092us) service_pool.cc:227] Handling call
0925 03:04:24.169528 (+ 76481us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:26.738596 (+2569068us) spinlock_profiling.cc:243] Waited 1.63 s on lock 0x7b54006412a0. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f4376460730 00007f43764d2300 00007f43764d3371 00007f43764cfedf 00007f43764cf99e 00007f437b4112e9 00007f4374af7684 00007f4374af7612 00007f4374af7541 00007f4374af74bc
0925 03:04:26.967410 (+228814us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:27.038509 (+ 71099us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3738434432}
I20260925 03:04:27.155412 25809 leader_election.cc:288] T 43366a0a2f9d4344bf4e37ffd290ef8a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:27.157073 24465 raft_consensus.cc:1226] T 9e8af2641e124f9589c5ccfdac785f63 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 2. Current term is 3. Ops: []
I20260925 03:04:27.138851 25761 raft_consensus.cc:482] T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:27.122244 25819 consensus_queue.cc:1035] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.027s
W20260925 03:04:27.164760 24436 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.718s user 0.018s sys 0.060s
I20260925 03:04:27.164887 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e5819c15eac4544a57e5ee6c9d4f117" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
W20260925 03:04:27.165758 24436 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.719s user 0.018s sys 0.060s
I20260925 03:04:27.147791 24616 rpcz_store.cc:276] 0925 03:04:26.025317 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:27.146065 (+1120748us) service_pool.cc:227] Handling call
0925 03:04:27.146412 (+ 347us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:27.147189 (+ 777us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:27.147396 (+ 207us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:27.163690 25761 raft_consensus.cc:3037] T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:27.172657 24467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5f9b503b0ab43999d582015a367441e" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:27.173985 24467 raft_consensus.cc:2379] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 3: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:04:27.178712 25741 leader_election.cc:288] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:27.180088 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7691c6b475428c82d6181d693d26f3" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:27.180526 24465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bb46bce3564841b7674e6a693bd2e6" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:27.180900 24468 raft_consensus.cc:2379] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 3: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:04:27.180721 24616 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38570 (request call id 1168) took 1116 ms. Trace:
I20260925 03:04:27.182196 24616 rpcz_store.cc:276] 0925 03:04:26.064206 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:27.179840 (+1115634us) service_pool.cc:227] Handling call
0925 03:04:27.180183 (+ 343us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:27.180365 (+ 182us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:27.180392 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:27.180438 (+ 46us) raft_consensus.cc:1631] Marking committed up to 1
0925 03:04:27.180469 (+ 31us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:27.180532 (+ 63us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:27.180646 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:27.183859 24616 raft_consensus.cc:2932] T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:27.181774 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "550118bc1a0641ce9803bdbf43f2069f" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:27.185300 25737 leader_election.cc:288] T 3d89396576884f7c8901dba54f783947 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:27.213611 24467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25102ba08e4e49388f6176badcd10a25" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:27.217239 24467 raft_consensus.cc:2454] T 25102ba08e4e49388f6176badcd10a25 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 1.
I20260925 03:04:27.218035 25845 raft_consensus.cc:482] T 9e8af2641e124f9589c5ccfdac785f63 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:27.218282 24467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6b6ec010caf4689b8c90449764f5cf9" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:27.218577 25845 raft_consensus.cc:504] T 9e8af2641e124f9589c5ccfdac785f63 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:27.220592 25817 raft_consensus.cc:3037] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:27.220624 25845 leader_election.cc:288] T 9e8af2641e124f9589c5ccfdac785f63 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:27.222676 25844 leader_election.cc:288] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
W20260925 03:04:27.224965 25804 consensus_meta.cc:287] T f22ddd7d1c444dc58f45a1fb0b3e02bc P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent flushing consensus metadata: real 0.559s user 0.004s sys 0.005s
I20260925 03:04:27.225414 25804 raft_consensus.cc:504] T f22ddd7d1c444dc58f45a1fb0b3e02bc P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:27.227543 25804 leader_election.cc:288] T f22ddd7d1c444dc58f45a1fb0b3e02bc P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:27.229291 24547 raft_consensus.cc:2379] T f22ddd7d1c444dc58f45a1fb0b3e02bc P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 in current term 4: Already voted for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in this term.
I20260925 03:04:27.115552 24472 rpcz_store.cc:276] 0925 03:04:24.108520 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:26.254765 (+2146245us) service_pool.cc:227] Handling call
0925 03:04:26.285520 (+ 30755us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:26.285725 (+ 205us) raft_consensus.cc:1520] Early marking committed up to index 5
0925 03:04:26.684721 (+398996us) log.cc:842] Serialized 14 byte log entry
0925 03:04:27.097218 (+412497us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:27.097278 (+ 60us) raft_consensus.cc:1631] Marking committed up to 5
0925 03:04:27.097325 (+ 47us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:27.097405 (+ 80us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:27.097546 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":210,"prepare.queue_time_us":559995,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":34,"raft.queue_time_us":5083,"raft.run_cpu_time_us":155,"raft.run_wall_time_us":6678,"thread_start_us":792124,"threads_started":2}
I20260925 03:04:27.227403 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5aec38e66a947c681e6c2a72b05e6ca" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:27.231762 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e725b82985746b39c0cc8f671c95353" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:27.232816 24468 raft_consensus.cc:2454] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 3.
I20260925 03:04:27.233083 24615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9357d144aa49789faaf31fe5e6bf89" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:27.236449 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60d45b7d467d4e1ab92367f6d7d52b15" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
W20260925 03:04:27.171162 25796 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.726s user 0.007s sys 0.014s
I20260925 03:04:27.249130 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37a157996e944db2906110d87ecb0c54" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:27.202203 24436 leader_election.cc:302] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
W20260925 03:04:27.253930 24436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 51799us
I20260925 03:04:27.234977 24615 raft_consensus.cc:2454] T ac9357d144aa49789faaf31fe5e6bf89 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 1.
W20260925 03:04:27.255936 24615 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38564 (request call id 1137) took 1324 ms (client timeout 1676 ms). Trace:
W20260925 03:04:27.256429 24615 rpcz_store.cc:269] 0925 03:04:25.931294 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:25.931418 (+ 124us) service_pool.cc:227] Handling call
0925 03:04:27.232934 (+1301516us) spinlock_profiling.cc:243] Waited 1.26 ms on lock 0x7b58001d3d98. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f43733ecf11 00007f4373570851 00007f4373570702 00007f437c460b09 00007f437b39c730 00007f437b46279b 00007f437b462821 00007f437b4121b0 00007f437b411a94 00007f4374af9c04 00007f4374af9b92
0925 03:04:27.255878 (+ 22944us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2893568}
I20260925 03:04:27.291031 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1414dcfe53f64a78ae413531eb8ec117" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:27.182736 25195 raft_consensus.cc:1226] T 04e068a0f80941dc9138cee063f53bab P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 LEADER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 1. Current term is 2. Ops: []
I20260925 03:04:27.302393 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c23337a7cb74184a0e57b7095cb6a11" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:27.303273 24545 raft_consensus.cc:2454] T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 3.
I20260925 03:04:27.254493 24467 raft_consensus.cc:2454] T e6b6ec010caf4689b8c90449764f5cf9 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 3.
W20260925 03:04:27.294147 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 194220us
I20260925 03:04:27.295014 24436 leader_election.cc:302] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: 1bc1f2e62ab44af995165bd5116a9d43, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:27.297497 25637 raft_consensus.cc:686] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c [term 3 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:27.297322 24464 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55560 (request call id 1120) took 3186 ms. Trace:
I20260925 03:04:27.305058 24436 leader_election.cc:302] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters: 1bc1f2e62ab44af995165bd5116a9d43
I20260925 03:04:27.305037 24464 rpcz_store.cc:276] 0925 03:04:24.110931 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:26.388457 (+2277526us) service_pool.cc:227] Handling call
0925 03:04:26.388844 (+ 387us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:26.389057 (+ 213us) raft_consensus.cc:1520] Early marking committed up to index 2
0925 03:04:27.117200 (+728143us) log.cc:842] Serialized 14 byte log entry
0925 03:04:27.296984 (+179784us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:27.297034 (+ 50us) raft_consensus.cc:1631] Marking committed up to 2
0925 03:04:27.297072 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:27.297136 (+ 64us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:27.297251 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":456860,"prepare.run_cpu_time_us":27,"prepare.run_wall_time_us":27,"raft.queue_time_us":94,"raft.run_cpu_time_us":92,"raft.run_wall_time_us":93,"thread_start_us":888594,"threads_started":2}
I20260925 03:04:27.312534 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06f4a7eaf6534e4986c9fd35911038e4" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:27.315981 24464 raft_consensus.cc:2379] T 06f4a7eaf6534e4986c9fd35911038e4 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 3: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
W20260925 03:04:27.249229 25796 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.839s user 0.007s sys 0.015s
I20260925 03:04:27.328284 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71f461a6e2ff434993b78a2d077f0abd" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:27.334160 24615 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38570 (request call id 1170) took 1226 ms. Trace:
I20260925 03:04:27.334573 24615 rpcz_store.cc:276] 0925 03:04:26.108248 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:27.257255 (+1149007us) service_pool.cc:227] Handling call
0925 03:04:27.333665 (+ 76410us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:27.333856 (+ 191us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:27.333880 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:27.333918 (+ 38us) raft_consensus.cc:1631] Marking committed up to 1
0925 03:04:27.333951 (+ 33us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:27.334004 (+ 53us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:27.334106 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:27.336895 24464 raft_consensus.cc:1226] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 2. Current term is 4. Ops: [2.5-2.5]
I20260925 03:04:27.337200 24467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21c43af5e6844fdaacc8c625fa01fd8d" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:27.337831 24588 leader_election.cc:302] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters: 1bc1f2e62ab44af995165bd5116a9d43
W20260925 03:04:27.254027 24466 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.803s user 0.011s sys 0.023s
I20260925 03:04:27.338976 24436 leader_election.cc:302] T 4e5332a5d329493991dff994e70ea92c P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:27.339681 24436 leader_election.cc:302] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:27.340523 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b5df303cd6a4d59aca5c18bbbff24ed" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:27.291883 25894 raft_consensus.cc:2781] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260925 03:04:27.347121 24436 leader_election.cc:302] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:27.348320 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:27.345994 24472 raft_consensus.cc:2379] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 3: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
W20260925 03:04:27.341517 24466 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.890s user 0.012s sys 0.024s
I20260925 03:04:27.183974 25851 raft_consensus.cc:504] T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:27.353106 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442d4a29b1d743599f4b6b4871e26668" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:27.353400 24436 leader_election.cc:302] T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:27.355787 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43366a0a2f9d4344bf4e37ffd290ef8a" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:27.333578 25783 raft_consensus.cc:2726] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260925 03:04:27.360742 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be4e0aa984f4f2b81a58e382e434ad5" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
W20260925 03:04:27.482352 24511 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 89428.5us
W20260925 03:04:27.469498 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 131734us
I20260925 03:04:27.432976 24470 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:55552 (request call id 436) took 2465 ms. Trace:
I20260925 03:04:27.340996 25844 raft_consensus.cc:2781] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260925 03:04:27.390343 25741 raft_consensus.cc:2781] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 03:04:27.501597 24470 rpcz_store.cc:276] 0925 03:04:24.967585 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:26.703470 (+1735885us) service_pool.cc:227] Handling call
0925 03:04:27.391521 (+688051us) spinlock_profiling.cc:243] Waited 143 ms on lock 0x7b5400141254. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455af0 00007f43764d8f2a 00007f437b415c6e 00007f4374b0c807 00007f4374b0c792 00007f4374b0c6c1 00007f4374b0c63c 00007f4374b0b8f2 00007f43748fa94c 00007f43748f9d86 00007f43748f972f
0925 03:04:27.432894 (+ 41373us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":329610496}
W20260925 03:04:27.483912 24436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 130593us
I20260925 03:04:27.483433 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:27.344965 25845 raft_consensus.cc:2781] T 4e5332a5d329493991dff994e70ea92c P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260925 03:04:27.487869 24586 connection.cc:232] RPC call timeout handler was delayed by -1.6617682619998959s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1504s for a response
I20260925 03:04:27.391114 25788 raft_consensus.cc:2781] T 7e49248643df4131a1c42e74244e05c2 P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260925 03:04:27.439980 25918 raft_consensus.cc:2781] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260925 03:04:27.502712 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7993f9c3e2a44f408bbf5d1e395e2306" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:27.512984 24622 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38570 (request call id 1165) took 1479 ms. Trace:
I20260925 03:04:27.363153 24467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6b6ec010caf4689b8c90449764f5cf9" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:27.519517 24467 raft_consensus.cc:2379] T e6b6ec010caf4689b8c90449764f5cf9 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 3: Already voted for candidate 1bc1f2e62ab44af995165bd5116a9d43 in this term.
I20260925 03:04:27.479827 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "420163a6cdce412ebc21c25fe3835638" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:27.507741 24614 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38564 (request call id 1142) took 1494 ms. Trace:
W20260925 03:04:27.515650 24586 connection.cc:232] RPC call timeout handler was delayed by -1.6365682619998552s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1532s for a response
I20260925 03:04:27.523411 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b627ecb4a5384ef392aa695e06599c39" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
W20260925 03:04:27.580328 24586 connection.cc:232] RPC call timeout handler was delayed by -1.6113682619998144s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15600000000000003s for a response
I20260925 03:04:27.439983 24539 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 1166) took 1025 ms. Trace:
W20260925 03:04:27.597435 24586 connection.cc:232] RPC call timeout handler was delayed by -1.5420682620006119s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1637s for a response
I20260925 03:04:27.516108 25741 raft_consensus.cc:482] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
W20260925 03:04:27.597859 24586 connection.cc:232] RPC call timeout handler was delayed by -1.4754682620005042s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17110000000000003s for a response
W20260925 03:04:27.598805 24586 connection.cc:232] RPC call timeout handler was delayed by -1.4727682619995903s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1714s for a response
W20260925 03:04:27.599144 24586 connection.cc:232] RPC call timeout handler was delayed by -1.4529682619995583s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1736s for a response
W20260925 03:04:27.599462 24586 connection.cc:232] RPC call timeout handler was delayed by -1.4403682619995379s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17500000000000002s for a response
I20260925 03:04:27.346271 25897 raft_consensus.cc:2781] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260925 03:04:27.519101 24622 rpcz_store.cc:276] 0925 03:04:26.033133 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:27.153184 (+1120051us) service_pool.cc:227] Handling call
0925 03:04:27.169623 (+ 16439us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:27.169833 (+ 210us) raft_consensus.cc:1520] Early marking committed up to index 3
0925 03:04:27.181176 (+ 11343us) log.cc:842] Serialized 14 byte log entry
0925 03:04:27.512617 (+331441us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:27.512671 (+ 54us) raft_consensus.cc:1631] Marking committed up to 3
0925 03:04:27.512733 (+ 62us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:27.512798 (+ 65us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:27.512924 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7829,"prepare.queue_time_us":11846,"prepare.run_cpu_time_us":25,"prepare.run_wall_time_us":25,"thread_start_us":303961,"threads_started":1}
I20260925 03:04:27.504947 25790 raft_consensus.cc:2781] T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260925 03:04:27.578034 24614 rpcz_store.cc:276] 0925 03:04:26.012965 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:26.654523 (+641558us) service_pool.cc:227] Handling call
0925 03:04:26.654857 (+ 334us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:26.655057 (+ 200us) raft_consensus.cc:1520] Early marking committed up to index 2
0925 03:04:27.073779 (+418722us) log.cc:842] Serialized 14 byte log entry
0925 03:04:27.507261 (+433482us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:27.507459 (+ 198us) raft_consensus.cc:1631] Marking committed up to 2
0925 03:04:27.507502 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:27.507569 (+ 67us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:27.507686 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":39861,"prepare.run_cpu_time_us":25,"prepare.run_wall_time_us":29,"raft.queue_time_us":143086,"raft.run_cpu_time_us":84,"raft.run_wall_time_us":84,"thread_start_us":825280,"threads_started":3,"wal-append.queue_time_us":353889}
I20260925 03:04:27.599607 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be4e0aa984f4f2b81a58e382e434ad5" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
W20260925 03:04:27.599929 24586 connection.cc:232] RPC call timeout handler was delayed by -1.4340682620004372s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1757s for a response
W20260925 03:04:27.608497 24586 connection.cc:232] RPC call timeout handler was delayed by -1.3134682620002422s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18910000000000002s for a response
W20260925 03:04:27.608924 24586 connection.cc:232] RPC call timeout handler was delayed by -1.2603682620010659s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.195s for a response
W20260925 03:04:27.609215 24586 connection.cc:232] RPC call timeout handler was delayed by -1.258568262001063s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1952s for a response
I20260925 03:04:27.506366 25844 raft_consensus.cc:482] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:27.596661 24588 leader_election.cc:302] T c88cbd39e01f44628a094af516be2833 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:27.519287 25845 raft_consensus.cc:482] T 4e5332a5d329493991dff994e70ea92c P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting leader election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:27.523568 25798 raft_consensus.cc:482] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
W20260925 03:04:27.604815 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 312266us
W20260925 03:04:27.609522 24586 connection.cc:232] RPC call timeout handler was delayed by -1.2558682620001491s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1955s for a response
I20260925 03:04:27.607946 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceab6b0fccd54e909a64f66fe81a9a1f" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:27.354110 25894 raft_consensus.cc:482] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting leader election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:27.359385 24618 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38570 (request call id 1166) took 1317 ms. Trace:
I20260925 03:04:27.390388 25190 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48408 (request call id 116) took 1356 ms. Trace:
I20260925 03:04:27.612363 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f1fac028a144058b92331c0d9b44ebf" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:27.621052 24472 raft_consensus.cc:2454] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 3.
I20260925 03:04:27.603719 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d89396576884f7c8901dba54f783947" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:27.522713 25842 consensus_queue.cc:1035] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.093s
I20260925 03:04:27.621927 24548 raft_consensus.cc:2454] T 3d89396576884f7c8901dba54f783947 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 1.
I20260925 03:04:27.519881 25885 raft_consensus.cc:482] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting leader election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:27.517102 25918 raft_consensus.cc:482] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting leader election (detected failure of leader 1bc1f2e62ab44af995165bd5116a9d43)
W20260925 03:04:27.618188 24586 connection.cc:232] RPC call timeout handler was delayed by -1.1748682620000181s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20450000000000002s for a response
I20260925 03:04:27.605157 25741 raft_consensus.cc:3037] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:27.611198 24545 raft_consensus.cc:2454] T 0be4e0aa984f4f2b81a58e382e434ad5 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 1.
W20260925 03:04:27.625411 24586 connection.cc:232] RPC call timeout handler was delayed by -1.1721682620009233s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2048s for a response
W20260925 03:04:27.625526 24514 leader_election.cc:334] T 8e5819c15eac4544a57e5ee6c9d4f117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187): Timed out: RequestConsensusVote RPC to 127.22.193.1:42187 timed out after 1.622s (SENT)
W20260925 03:04:27.626128 24586 connection.cc:232] RPC call timeout handler was delayed by -1.0839682620007807s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2146s for a response
I20260925 03:04:27.606614 25790 raft_consensus.cc:482] T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting leader election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:27.615062 25844 raft_consensus.cc:3037] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Advancing to term 4
W20260925 03:04:27.626816 24586 connection.cc:232] RPC call timeout handler was delayed by -1.0785682620007719s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21520000000000003s for a response
W20260925 03:04:27.616775 24436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 94498.3us
I20260925 03:04:27.627602 24622 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38570 (request call id 1174) took 1461 ms. Trace:
I20260925 03:04:27.603220 25897 raft_consensus.cc:482] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting leader election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:27.628363 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21c43af5e6844fdaacc8c625fa01fd8d" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:27.615480 25845 raft_consensus.cc:3037] T 4e5332a5d329493991dff994e70ea92c P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:27.634754 24617 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38564 (request call id 1144) took 1600 ms. Trace:
I20260925 03:04:27.626504 24514 leader_election.cc:302] T 8e5819c15eac4544a57e5ee6c9d4f117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:27.632182 24622 rpcz_store.cc:276] 0925 03:04:26.166410 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:27.610837 (+1444427us) service_pool.cc:227] Handling call
0925 03:04:27.627286 (+ 16449us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:27.627402 (+ 116us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:27.627417 (+ 15us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:27.627442 (+ 25us) raft_consensus.cc:1631] Marking committed up to 1
0925 03:04:27.627469 (+ 27us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:27.627500 (+ 31us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:27.627563 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:27.636802 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c88cbd39e01f44628a094af516be2833" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:27.635109 24617 rpcz_store.cc:276] 0925 03:04:26.034223 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:27.156380 (+1122157us) service_pool.cc:227] Handling call
0925 03:04:27.159183 (+ 2803us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:27.159421 (+ 238us) raft_consensus.cc:1520] Early marking committed up to index 2
0925 03:04:27.171954 (+ 12533us) log.cc:842] Serialized 14 byte log entry
0925 03:04:27.634491 (+462537us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:27.634532 (+ 41us) raft_consensus.cc:1631] Marking committed up to 2
0925 03:04:27.634560 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:27.634607 (+ 47us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:27.634705 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":22063,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":37,"raft.queue_time_us":115472,"raft.run_cpu_time_us":53,"raft.run_wall_time_us":54,"thread_start_us":462070,"threads_started":2,"wal-append.queue_time_us":180066}
I20260925 03:04:27.611348 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367277daf8d04778b3fa4ed94d71580d" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:27.629977 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7691c6b475428c82d6181d693d26f3" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
W20260925 03:04:27.631825 24586 connection.cc:232] RPC call timeout handler was delayed by -1.038068261999797s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2197s for a response
I20260925 03:04:27.622529 24540 raft_consensus.cc:2379] T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1bc1f2e62ab44af995165bd5116a9d43 in current term 2: Already voted for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in this term.
I20260925 03:04:27.438230 25637 consensus_queue.cc:237] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:27.630439 25918 raft_consensus.cc:3037] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:27.622710 25894 raft_consensus.cc:3037] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:27.646189 24622 raft_consensus.cc:1226] T 7993f9c3e2a44f408bbf5d1e395e2306 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 1. Current term is 2. Ops: []
I20260925 03:04:27.668485 24622 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38564 (request call id 1148) took 1326 ms. Trace:
I20260925 03:04:27.616214 25761 raft_consensus.cc:504] T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:27.647441 24617 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38570 (request call id 1176) took 1241 ms. Trace:
I20260925 03:04:27.652292 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b5df303cd6a4d59aca5c18bbbff24ed" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:27.620667 25798 raft_consensus.cc:3037] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:27.598019 24539 rpcz_store.cc:276] 0925 03:04:26.414845 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:26.706347 (+291502us) service_pool.cc:227] Handling call
0925 03:04:26.706707 (+ 360us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:27.126702 (+419995us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:27.132966 (+ 6264us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:27.137518 (+ 4552us) raft_consensus.cc:1631] Marking committed up to 1
0925 03:04:27.139818 (+ 2300us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:27.439719 (+299901us) spinlock_profiling.cc:243] Waited 412 ms on lock 0x7b54000131d4. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d0f51 00007f43764cf99e 00007f437b4112e9 00007f4374af7684 00007f4374af7612 00007f4374af7541 00007f4374af74bc 00007f4374af6772 00007f43748fa94c 00007f43748f9d86
0925 03:04:27.439781 (+ 62us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:27.439923 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":946668288}
W20260925 03:04:27.656015 25913 log.cc:924] Time spent T f22ddd7d1c444dc58f45a1fb0b3e02bc P 869b0a8575b54d08ae2817fad5dd42d5: Append to log took a long time: real 0.057s user 0.000s sys 0.000s
I20260925 03:04:27.627018 25885 raft_consensus.cc:3037] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:27.630800 25790 raft_consensus.cc:3037] T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:27.665946 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7aa47c48404465b94a36fd819fe3c7a" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
W20260925 03:04:27.667443 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 50905.2us
I20260925 03:04:27.668993 24622 rpcz_store.cc:276] 0925 03:04:26.342429 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:27.645638 (+1303209us) service_pool.cc:227] Handling call
0925 03:04:27.645991 (+ 353us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:27.668187 (+ 22196us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:27.668407 (+ 220us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260925 03:04:27.648175 24586 connection.cc:232] RPC call timeout handler was delayed by -1.0182682619997649s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2219s for a response
I20260925 03:04:27.670890 24614 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38564 (request call id 1146) took 1412 ms. Trace:
I20260925 03:04:27.670682 24545 raft_consensus.cc:2454] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 2.
I20260925 03:04:27.673259 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1414dcfe53f64a78ae413531eb8ec117" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:27.707942 24588 leader_election.cc:302] T 442d4a29b1d743599f4b6b4871e26668 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:27.637324 25916 raft_consensus.cc:2781] T c88cbd39e01f44628a094af516be2833 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260925 03:04:27.708900 25916 raft_consensus.cc:482] T c88cbd39e01f44628a094af516be2833 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting leader election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:27.709394 25916 raft_consensus.cc:3037] T c88cbd39e01f44628a094af516be2833 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:27.675740 24548 raft_consensus.cc:2454] T 0b5df303cd6a4d59aca5c18bbbff24ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 3.
I20260925 03:04:27.675410 24617 rpcz_store.cc:276] 0925 03:04:26.406507 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:27.646545 (+1240038us) service_pool.cc:227] Handling call
0925 03:04:27.646892 (+ 347us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:27.647073 (+ 181us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:27.647098 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:27.647143 (+ 45us) raft_consensus.cc:1631] Marking committed up to 1
0925 03:04:27.647175 (+ 32us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:27.647227 (+ 52us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:27.647381 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:27.717442 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:27.707428 24614 rpcz_store.cc:276] 0925 03:04:26.258333 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:27.612273 (+1353940us) service_pool.cc:227] Handling call
0925 03:04:27.612728 (+ 455us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:27.613014 (+ 286us) raft_consensus.cc:1520] Early marking committed up to index 2
0925 03:04:27.620235 (+ 7221us) log.cc:842] Serialized 14 byte log entry
0925 03:04:27.670537 (+ 50302us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:27.670591 (+ 54us) raft_consensus.cc:1631] Marking committed up to 2
0925 03:04:27.670634 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:27.670704 (+ 70us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:27.670830 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":8312,"prepare.run_cpu_time_us":28,"prepare.run_wall_time_us":28,"raft.queue_time_us":102,"raft.run_cpu_time_us":177,"raft.run_wall_time_us":18541,"thread_start_us":49870,"threads_started":1}
I20260925 03:04:27.720285 24614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7691c6b475428c82d6181d693d26f3" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:27.721226 24614 raft_consensus.cc:2454] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 3.
I20260925 03:04:27.686610 25741 raft_consensus.cc:504] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:27.636119 25897 raft_consensus.cc:3037] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Advancing to term 4
W20260925 03:04:27.658414 25908 log.cc:924] Time spent T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.051s user 0.000s sys 0.000s
I20260925 03:04:27.722661 24614 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38570 (request call id 1177) took 1306 ms. Trace:
I20260925 03:04:27.661134 24541 raft_consensus.cc:2454] T 367277daf8d04778b3fa4ed94d71580d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 3.
W20260925 03:04:27.726636 25165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 91483.1us
I20260925 03:04:27.727160 24614 rpcz_store.cc:276] 0925 03:04:26.416125 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:27.721956 (+1305831us) service_pool.cc:227] Handling call
0925 03:04:27.722225 (+ 269us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:27.722391 (+ 166us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:27.722415 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:27.722453 (+ 38us) raft_consensus.cc:1631] Marking committed up to 1
0925 03:04:27.722477 (+ 24us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:27.722523 (+ 46us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:27.722614 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:27.729632 24614 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34546 (request call id 50) took 1088 ms. Trace:
I20260925 03:04:27.730031 24614 rpcz_store.cc:276] 0925 03:04:26.641117 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:27.728798 (+1087681us) service_pool.cc:227] Handling call
0925 03:04:27.729078 (+ 280us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:27.729237 (+ 159us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:27.729260 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:27.729298 (+ 38us) raft_consensus.cc:1631] Marking committed up to 5
0925 03:04:27.729325 (+ 27us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:27.729380 (+ 55us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:27.729469 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260925 03:04:27.731853 24436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 83438.4us
I20260925 03:04:27.732559 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d89396576884f7c8901dba54f783947" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
W20260925 03:04:27.721493 24621 consensus_meta.cc:287] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c: Time spent flushing consensus metadata: real 0.571s user 0.010s sys 0.000s
I20260925 03:04:27.727231 25637 raft_consensus.cc:482] T b1e85248d3b04b979dd080caeeb02fdb P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:27.749842 25637 raft_consensus.cc:504] T b1e85248d3b04b979dd080caeeb02fdb P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:27.758235 25894 raft_consensus.cc:504] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:27.622834 24618 rpcz_store.cc:276] 0925 03:04:26.037267 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:27.158486 (+1121219us) service_pool.cc:227] Handling call
0925 03:04:27.212051 (+ 53565us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:27.257321 (+ 45270us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:27.290592 (+ 33271us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:27.301259 (+ 10667us) raft_consensus.cc:1631] Marking committed up to 1
0925 03:04:27.336585 (+ 35326us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:27.344003 (+ 7418us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:27.354438 (+ 10435us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:27.741569 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e8af2641e124f9589c5ccfdac785f63" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
W20260925 03:04:27.768756 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 99977.7us
I20260925 03:04:27.770707 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43366a0a2f9d4344bf4e37ffd290ef8a" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
W20260925 03:04:27.772382 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 64518.4us
I20260925 03:04:27.722757 24466 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 601) took 3423 ms. Trace:
W20260925 03:04:27.740732 25915 log.cc:924] Time spent T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.278s user 0.000s sys 0.001s
I20260925 03:04:27.697014 25786 consensus_queue.cc:1035] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.046s
I20260925 03:04:27.777639 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "550118bc1a0641ce9803bdbf43f2069f" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:27.777799 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4ebd2b043024a548005c8b6b00b0449" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:27.722184 25854 leader_election.cc:288] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:27.737489 25926 raft_consensus.cc:2781] T 442d4a29b1d743599f4b6b4871e26668 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260925 03:04:27.786301 24617 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38570 (request call id 1179) took 1302 ms. Trace:
I20260925 03:04:27.790299 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcf6e6a422d4d198d4fbb0f7d3d845d" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:27.742877 24621 raft_consensus.cc:2454] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 3.
I20260925 03:04:27.791391 24622 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38564 (request call id 1154) took 1097 ms. Trace:
I20260925 03:04:27.803027 24622 rpcz_store.cc:276] 0925 03:04:26.693822 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:27.774487 (+1080665us) service_pool.cc:227] Handling call
0925 03:04:27.790939 (+ 16452us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:27.791104 (+ 165us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:27.791131 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:27.791168 (+ 37us) raft_consensus.cc:1631] Marking committed up to 2
0925 03:04:27.791197 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:27.791244 (+ 47us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:27.791337 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:27.790783 24617 rpcz_store.cc:276] 0925 03:04:26.483601 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:27.723689 (+1240088us) service_pool.cc:227] Handling call
0925 03:04:27.723961 (+ 272us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:27.724166 (+ 205us) raft_consensus.cc:1520] Early marking committed up to index 2
0925 03:04:27.734140 (+ 9974us) log.cc:842] Serialized 14 byte log entry
0925 03:04:27.785996 (+ 51856us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:27.786049 (+ 53us) raft_consensus.cc:1631] Marking committed up to 2
0925 03:04:27.786089 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:27.786152 (+ 63us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:27.786251 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":10225,"prepare.run_cpu_time_us":60,"prepare.run_wall_time_us":1477,"thread_start_us":51558,"threads_started":1}
I20260925 03:04:27.805794 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43366a0a2f9d4344bf4e37ffd290ef8a" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:27.806566 24617 raft_consensus.cc:2454] T 43366a0a2f9d4344bf4e37ffd290ef8a P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 1.
I20260925 03:04:27.809082 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f22ddd7d1c444dc58f45a1fb0b3e02bc" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:27.811018 24615 raft_consensus.cc:1226] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 2. Current term is 3. Ops: []
I20260925 03:04:27.799434 25897 raft_consensus.cc:504] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:27.766831 25637 leader_election.cc:288] T b1e85248d3b04b979dd080caeeb02fdb P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:27.818014 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7aa47c48404465b94a36fd819fe3c7a" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:27.825712 24471 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 596) took 3647 ms. Trace:
I20260925 03:04:27.830464 24471 rpcz_store.cc:276] 0925 03:04:24.178422 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:26.389216 (+2210794us) service_pool.cc:227] Handling call
0925 03:04:26.403971 (+ 14755us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:04:27.046919 (+642948us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:27.046954 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:04:27.092217 (+ 45263us) log.cc:842] Serialized 26 byte log entry
0925 03:04:27.297983 (+205766us) raft_consensus.cc:1631] Marking committed up to 1
0925 03:04:27.298026 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:27.298096 (+ 70us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:04:27.825421 (+527325us) raft_consensus.cc:1671] finished
0925 03:04:27.825445 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:27.825620 (+ 175us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3096,"raft.queue_time_us":43836,"raft.run_cpu_time_us":202,"raft.run_wall_time_us":33527,"thread_start_us":764046,"threads_started":2,"wal-append.queue_time_us":365534}
W20260925 03:04:27.808615 25906 log.cc:924] Time spent T 7e49248643df4131a1c42e74244e05c2 P 1bc1f2e62ab44af995165bd5116a9d43: Append to log took a long time: real 0.210s user 0.000s sys 0.001s
I20260925 03:04:27.827397 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d89396576884f7c8901dba54f783947" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:27.790241 25911 raft_consensus.cc:482] T c85f5d1d3f004cf5817bb594fd8e6469 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:27.790192 25926 raft_consensus.cc:482] T 442d4a29b1d743599f4b6b4871e26668 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting leader election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:27.832309 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcf6e6a422d4d198d4fbb0f7d3d845d" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:27.801613 25851 leader_election.cc:288] T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 election: Requested vote from peers eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:27.808482 24621 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38564 (request call id 1159) took 1046 ms. Trace:
W20260925 03:04:27.843523 25165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 103719us
W20260925 03:04:27.855008 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 145418us
I20260925 03:04:27.857779 24586 leader_election.cc:302] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:04:27.859613 24586 leader_election.cc:302] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 1bc1f2e62ab44af995165bd5116a9d43, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:27.848412 24621 rpcz_store.cc:276] 0925 03:04:26.761716 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:27.045963 (+284247us) service_pool.cc:227] Handling call
0925 03:04:27.803728 (+757765us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:27.862026 24586 leader_election.cc:302] T 550118bc1a0641ce9803bdbf43f2069f P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5
W20260925 03:04:27.854203 24436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 117253us
I20260925 03:04:27.864637 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc2df40db08649c9a75e90000b878559" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:27.866140 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f22ddd7d1c444dc58f45a1fb0b3e02bc" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:27.836468 25926 raft_consensus.cc:3037] T 442d4a29b1d743599f4b6b4871e26668 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:27.855443 25197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:27.869239 25197 raft_consensus.cc:2454] T 29637f5a4a504ce6ba29033ccd0cd716 P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 3.
I20260925 03:04:27.870388 25923 raft_consensus.cc:2781] T 550118bc1a0641ce9803bdbf43f2069f P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 03:04:27.871016 25923 raft_consensus.cc:482] T 550118bc1a0641ce9803bdbf43f2069f P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting leader election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:27.871119 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7691c6b475428c82d6181d693d26f3" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:27.871490 25923 raft_consensus.cc:3037] T 550118bc1a0641ce9803bdbf43f2069f P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:27.871727 24472 raft_consensus.cc:2454] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 3.
I20260925 03:04:27.872439 24616 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38570 (request call id 1169) took 1789 ms. Trace:
I20260925 03:04:27.872642 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06f4a7eaf6534e4986c9fd35911038e4" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:27.873207 24472 raft_consensus.cc:3037] T 06f4a7eaf6534e4986c9fd35911038e4 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:27.873034 24616 rpcz_store.cc:276] 0925 03:04:26.082823 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:27.183267 (+1100444us) service_pool.cc:227] Handling call
0925 03:04:27.183534 (+ 267us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:27.183741 (+ 207us) raft_consensus.cc:1520] Early marking committed up to index 4
0925 03:04:27.661297 (+477556us) log.cc:842] Serialized 14 byte log entry
0925 03:04:27.837078 (+175781us) log.cc:842] Serialized 14 byte log entry
0925 03:04:27.858564 (+ 21486us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:27.871938 (+ 13374us) raft_consensus.cc:1631] Marking committed up to 4
0925 03:04:27.872032 (+ 94us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:27.872160 (+ 128us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:27.872350 (+ 190us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11774,"prepare.queue_time_us":36320,"prepare.run_wall_time_us":41,"raft.queue_time_us":10319,"raft.run_cpu_time_us":356,"raft.run_wall_time_us":34437,"thread_start_us":90504,"threads_started":1}
I20260925 03:04:27.870738 25925 raft_consensus.cc:2781] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260925 03:04:27.876168 25925 raft_consensus.cc:482] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Starting leader election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:27.876596 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b5df303cd6a4d59aca5c18bbbff24ed" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:27.878222 24471 raft_consensus.cc:2454] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 3.
I20260925 03:04:27.880342 24436 leader_election.cc:302] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:27.876524 25925 raft_consensus.cc:3037] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:27.882900 25637 consensus_queue.cc:1035] T 11bb46bce3564841b7674e6a693bd2e6 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260925 03:04:27.775436 24466 rpcz_store.cc:276] 0925 03:04:24.299743 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:26.450113 (+2150370us) service_pool.cc:227] Handling call
0925 03:04:26.450505 (+ 392us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:26.450742 (+ 237us) raft_consensus.cc:1520] Early marking committed up to index 2
0925 03:04:27.448796 (+998054us) log.cc:842] Serialized 14 byte log entry
0925 03:04:27.661656 (+212860us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:27.668483 (+ 6827us) raft_consensus.cc:1631] Marking committed up to 2
0925 03:04:27.672852 (+ 4369us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:27.691207 (+ 18355us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:27.707613 (+ 16406us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":395331,"prepare.run_cpu_time_us":22,"prepare.run_wall_time_us":23,"raft.queue_time_us":6596,"raft.run_cpu_time_us":104,"raft.run_wall_time_us":106,"thread_start_us":1091492,"threads_started":2}
I20260925 03:04:27.882769 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43366a0a2f9d4344bf4e37ffd290ef8a" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
W20260925 03:04:27.859714 25919 log.cc:924] Time spent T 367277daf8d04778b3fa4ed94d71580d P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.194s user 0.000s sys 0.001s
I20260925 03:04:27.886247 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1e85248d3b04b979dd080caeeb02fdb" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
W20260925 03:04:27.886799 24468 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.543s user 0.016s sys 0.013s
I20260925 03:04:27.852705 25938 raft_consensus.cc:1050] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5: attempting to promote NON_VOTER 1bc1f2e62ab44af995165bd5116a9d43 to VOTER
W20260925 03:04:27.890852 24468 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.547s user 0.016s sys 0.013s
I20260925 03:04:27.851608 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442d4a29b1d743599f4b6b4871e26668" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:27.862159 25761 leader_election.cc:288] T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:27.865232 25931 raft_consensus.cc:2726] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260925 03:04:27.890409 25920 log.cc:924] Time spent T c85f5d1d3f004cf5817bb594fd8e6469 P 869b0a8575b54d08ae2817fad5dd42d5: Append to log took a long time: real 0.250s user 0.000s sys 0.001s
I20260925 03:04:27.839570 25911 raft_consensus.cc:504] T c85f5d1d3f004cf5817bb594fd8e6469 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:27.622933 25190 rpcz_store.cc:276] 0925 03:04:26.033630 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:26.124083 (+ 90453us) service_pool.cc:227] Handling call
0925 03:04:26.163654 (+ 39571us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:26.163866 (+ 212us) raft_consensus.cc:1520] Early marking committed up to index 4
0925 03:04:26.772860 (+608994us) log.cc:842] Serialized 14 byte log entry
0925 03:04:27.350492 (+577632us) log.cc:842] Serialized 14 byte log entry
0925 03:04:27.389903 (+ 39411us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:27.389968 (+ 65us) raft_consensus.cc:1631] Marking committed up to 4
0925 03:04:27.390018 (+ 50us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:27.390114 (+ 96us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:27.390295 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":37741,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":36,"raft.queue_time_us":71101,"raft.run_cpu_time_us":237,"raft.run_wall_time_us":25412,"thread_start_us":461980,"threads_started":1,"wal-append.queue_time_us":709559}
I20260925 03:04:27.870093 25942 consensus_queue.cc:1035] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.031s
W20260925 03:04:27.924203 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 60521.4us
I20260925 03:04:27.931933 24586 leader_election.cc:302] T 04e068a0f80941dc9138cee063f53bab P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 1bc1f2e62ab44af995165bd5116a9d43, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:27.893997 25192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1e85248d3b04b979dd080caeeb02fdb" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:27.935202 24586 leader_election.cc:302] T 74443a0a1dc34d88a2f42e3b722db405 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:27.896577 24618 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38564 (request call id 1156) took 1135 ms. Trace:
I20260925 03:04:27.943634 24586 leader_election.cc:302] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
W20260925 03:04:27.943639 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 74803us
I20260925 03:04:27.947610 24588 leader_election.cc:302] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:27.934615 25165 leader_election.cc:302] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5; no voters: eeab93abfdb141a895002ec9f312296c
W20260925 03:04:27.931955 24436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 49994us
I20260925 03:04:27.948225 24472 raft_consensus.cc:2454] T 06f4a7eaf6534e4986c9fd35911038e4 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 4.
I20260925 03:04:27.947914 24586 leader_election.cc:302] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:04:27.944646 25192 raft_consensus.cc:2454] T b1e85248d3b04b979dd080caeeb02fdb P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 1.
I20260925 03:04:27.955410 24588 leader_election.cc:302] T 0b5df303cd6a4d59aca5c18bbbff24ed P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:27.956318 24586 leader_election.cc:302] T cc2df40db08649c9a75e90000b878559 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:04:27.957183 25165 leader_election.cc:302] T e6b6ec010caf4689b8c90449764f5cf9 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:27.939761 25741 leader_election.cc:288] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
W20260925 03:04:27.947714 25817 consensus_meta.cc:287] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent flushing consensus metadata: real 0.727s user 0.012s sys 0.000s
I20260925 03:04:27.941009 25761 raft_consensus.cc:2726] T 04e068a0f80941dc9138cee063f53bab P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 03:04:27.935818 25894 leader_election.cc:288] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:27.937266 25197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15861cd28e354598a3c6bfc4337c34bb" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43"
I20260925 03:04:27.940618 24618 rpcz_store.cc:276] 0925 03:04:26.760932 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:27.783820 (+1022888us) service_pool.cc:227] Handling call
0925 03:04:27.834122 (+ 50302us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:27.850439 (+ 16317us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:27.852786 (+ 2347us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:27.859370 (+ 6584us) raft_consensus.cc:1631] Marking committed up to 4
0925 03:04:27.866191 (+ 6821us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:27.884628 (+ 18437us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:27.893079 (+ 8451us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:27.934435 24616 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38564 (request call id 1161) took 1172 ms. Trace:
W20260925 03:04:27.940297 24613 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38570 (request call id 1171) took 1813 ms (client timeout 2191 ms). Trace:
W20260925 03:04:27.993643 25935 log.cc:924] Time spent T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.160s user 0.000s sys 0.002s
I20260925 03:04:27.998205 25909 raft_consensus.cc:2781] T e6b6ec010caf4689b8c90449764f5cf9 P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 03:04:27.998136 25922 consensus_queue.cc:1035] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.054s
I20260925 03:04:27.947521 25931 raft_consensus.cc:2726] T 74443a0a1dc34d88a2f42e3b722db405 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260925 03:04:28.006506 24546 raft_consensus.cc:2396] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 for term 4 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260925 03:04:28.037580 25909 raft_consensus.cc:686] T e6b6ec010caf4689b8c90449764f5cf9 P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 LEADER]: Becoming Leader. State: Replica: 1bc1f2e62ab44af995165bd5116a9d43, State: Running, Role: LEADER
I20260925 03:04:28.038899 25931 raft_consensus.cc:2781] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260925 03:04:27.962283 25936 raft_consensus.cc:2781] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260925 03:04:28.045723 25909 consensus_queue.cc:237] T e6b6ec010caf4689b8c90449764f5cf9 P 1bc1f2e62ab44af995165bd5116a9d43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:28.045476 25922 raft_consensus.cc:2781] T 0b5df303cd6a4d59aca5c18bbbff24ed P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260925 03:04:28.051335 25922 raft_consensus.cc:482] T 0b5df303cd6a4d59aca5c18bbbff24ed P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:28.052027 25922 raft_consensus.cc:3037] T 0b5df303cd6a4d59aca5c18bbbff24ed P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Advancing to term 4
W20260925 03:04:28.015833 24469 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38920 (request call id 56) took 1977 ms (client timeout 1875 ms). Trace:
W20260925 03:04:28.052932 24469 rpcz_store.cc:269] 0925 03:04:26.038364 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:26.132229 (+ 93865us) service_pool.cc:227] Handling call
0925 03:04:28.015488 (+1883259us) spinlock_profiling.cc:243] Waited 252 ms on lock 0x7b54001919d4. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d42ae 00007f437b411c27 00007f4374af9c04 00007f4374af9b92 00007f4374af9ac1 00007f4374af9a3c 00007f4374af8cf2 00007f43748fa94c 00007f43748f9d86 00007f43748f972f
0925 03:04:28.015698 (+ 210us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":579592960}
I20260925 03:04:28.031101 25197 raft_consensus.cc:3037] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 FOLLOWER]: Advancing to term 4
W20260925 03:04:28.019383 24613 rpcz_store.cc:269] 0925 03:04:26.124619 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:26.180991 (+ 56372us) service_pool.cc:227] Handling call
0925 03:04:27.838784 (+1657793us) spinlock_profiling.cc:243] Waited 3.1 ms on lock 0x7b58001d3d98. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f43733ecf11 00007f4373570851 00007f4373570702 00007f437c460b09 00007f437b39c730 00007f437b46279b 00007f437b462821 00007f437b4121b0 00007f437b411a94 00007f4374af9c04 00007f4374af9b92
0925 03:04:27.937436 (+ 98652us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7133568}
W20260925 03:04:28.050441 25870 log.cc:924] Time spent T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5: Append to log took a long time: real 1.655s user 0.003s sys 0.007s
I20260925 03:04:28.039624 25761 raft_consensus.cc:2781] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260925 03:04:28.081336 25761 raft_consensus.cc:482] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:28.082029 25761 raft_consensus.cc:3037] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:28.082698 25844 raft_consensus.cc:504] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:28.076480 25897 leader_election.cc:288] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:27.998301 25933 raft_consensus.cc:2781] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260925 03:04:28.042697 25931 raft_consensus.cc:482] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting leader election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:28.048825 25936 raft_consensus.cc:482] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260925 03:04:28.091346 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 135994us
I20260925 03:04:27.998493 25845 raft_consensus.cc:504] T 4e5332a5d329493991dff994e70ea92c P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:28.110402 25951 raft_consensus.cc:2781] T cc2df40db08649c9a75e90000b878559 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 03:04:28.111505 25951 raft_consensus.cc:482] T cc2df40db08649c9a75e90000b878559 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting leader election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:28.112457 25951 raft_consensus.cc:3037] T cc2df40db08649c9a75e90000b878559 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:28.082043 25647 raft_consensus.cc:3037] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:28.033735 24473 raft_consensus.cc:2932] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:28.128710 24588 leader_election.cc:302] T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:28.035024 24472 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0be4e0aa984f4f2b81a58e382e434ad5"
dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
mode: GRACEFUL
new_leader_uuid: "eeab93abfdb141a895002ec9f312296c"
from {username='slave'} at 127.0.0.1:38832
I20260925 03:04:28.129983 24472 raft_consensus.cc:595] T 0be4e0aa984f4f2b81a58e382e434ad5 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 LEADER]: Received request to transfer leadership to TS eeab93abfdb141a895002ec9f312296c
I20260925 03:04:28.132092 24467 raft_consensus.cc:1226] T 4e5332a5d329493991dff994e70ea92c P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 3. Current term is 4. Ops: []
W20260925 03:04:28.134701 25948 log.cc:924] Time spent T 37a157996e944db2906110d87ecb0c54 P 869b0a8575b54d08ae2817fad5dd42d5: Append to log took a long time: real 0.060s user 0.000s sys 0.001s
I20260925 03:04:28.135185 25916 raft_consensus.cc:504] T c88cbd39e01f44628a094af516be2833 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:28.102082 25931 raft_consensus.cc:3037] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:28.137981 24472 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0e54ca1757ee4e42956a9cebbeca4f62"
dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
mode: GRACEFUL
new_leader_uuid: "eeab93abfdb141a895002ec9f312296c"
from {username='slave'} at 127.0.0.1:38832
I20260925 03:04:28.142417 24472 raft_consensus.cc:595] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Received request to transfer leadership to TS eeab93abfdb141a895002ec9f312296c
I20260925 03:04:28.142789 24472 raft_consensus.cc:603] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260925 03:04:28.138825 25945 log.cc:924] Time spent T 442d4a29b1d743599f4b6b4871e26668 P 869b0a8575b54d08ae2817fad5dd42d5: Append to log took a long time: real 0.097s user 0.000s sys 0.001s
I20260925 03:04:28.109158 25936 raft_consensus.cc:3037] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:28.019513 24616 rpcz_store.cc:276] 0925 03:04:26.762271 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:27.876126 (+1113855us) service_pool.cc:227] Handling call
0925 03:04:27.876664 (+ 538us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:27.891353 (+ 14689us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:27.891381 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:27.894149 (+ 2768us) raft_consensus.cc:1631] Marking committed up to 2
0925 03:04:27.904506 (+ 10357us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:27.914749 (+ 10243us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:27.931140 (+ 16391us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:28.146270 25845 leader_election.cc:288] T 4e5332a5d329493991dff994e70ea92c P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:28.123432 25933 raft_consensus.cc:482] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:28.159618 25844 leader_election.cc:288] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 election: Requested vote from peers eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:28.159795 25933 raft_consensus.cc:3037] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:28.163131 24472 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1414dcfe53f64a78ae413531eb8ec117"
dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
mode: GRACEFUL
new_leader_uuid: "eeab93abfdb141a895002ec9f312296c"
from {username='slave'} at 127.0.0.1:38832
I20260925 03:04:28.163805 24472 raft_consensus.cc:595] T 1414dcfe53f64a78ae413531eb8ec117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 LEADER]: Received request to transfer leadership to TS eeab93abfdb141a895002ec9f312296c
I20260925 03:04:28.041227 25938 consensus_queue.cc:237] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 2, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:28.187793 25636 raft_consensus.cc:2781] T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader election won for term 4
I20260925 03:04:28.192116 25636 raft_consensus.cc:686] T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c [term 4 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:28.200026 25636 consensus_queue.cc:237] T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:28.201124 25936 raft_consensus.cc:504] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:28.211206 25788 raft_consensus.cc:482] T 7e49248643df4131a1c42e74244e05c2 P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:28.215234 24619 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38564 (request call id 1141) took 2211 ms. Trace:
I20260925 03:04:28.218356 25911 leader_election.cc:288] T c85f5d1d3f004cf5817bb594fd8e6469 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
W20260925 03:04:28.229776 25765 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.003s user 0.023s sys 0.015s
W20260925 03:04:28.231774 25765 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.005s user 0.024s sys 0.015s
W20260925 03:04:28.231144 24436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 273762us
I20260925 03:04:28.222678 24619 rpcz_store.cc:276] 0925 03:04:26.004031 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:26.647358 (+643327us) service_pool.cc:227] Handling call
0925 03:04:27.053056 (+405698us) spinlock_profiling.cc:243] Waited 1.91 ms on lock 0x7b58001d3d98. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f43733ecf11 00007f4373570851 00007f4373570702 00007f437c460b09 00007f437b39c730 00007f437b46279b 00007f437b462821 00007f437b4262b0 00007f437b4117f9 00007f437b4112a2 00007f4374af7684
0925 03:04:27.053254 (+ 198us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:27.118050 (+ 64796us) raft_consensus.cc:1520] Early marking committed up to index 2
0925 03:04:27.131917 (+ 13867us) log.cc:842] Serialized 14 byte log entry
0925 03:04:27.517739 (+385822us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:27.517789 (+ 50us) raft_consensus.cc:1631] Marking committed up to 2
0925 03:04:27.517843 (+ 54us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:28.214991 (+697148us) spinlock_profiling.cc:243] Waited 64.5 ms on lock 0x7b54002eefd4. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d0f51 00007f43764cf99e 00007f437b4112e9 00007f4374af7684 00007f4374af7612 00007f4374af7541 00007f4374af74bc 00007f4374af6772 00007f43748fa94c 00007f43748f9d86
0925 03:04:28.215048 (+ 57us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:28.215177 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":16350,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":39,"raft.queue_time_us":115222,"raft.run_cpu_time_us":52,"raft.run_wall_time_us":53,"spinlock_wait_cycles":152810112,"thread_start_us":385428,"threads_started":2,"wal-append.queue_time_us":227601}
I20260925 03:04:28.236202 24472 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3d89396576884f7c8901dba54f783947"
dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
mode: GRACEFUL
new_leader_uuid: "eeab93abfdb141a895002ec9f312296c"
from {username='slave'} at 127.0.0.1:38832
I20260925 03:04:28.236765 24472 raft_consensus.cc:595] T 3d89396576884f7c8901dba54f783947 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 LEADER]: Received request to transfer leadership to TS eeab93abfdb141a895002ec9f312296c
W20260925 03:04:28.237721 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 281469us
I20260925 03:04:28.238108 24619 raft_consensus.cc:1261] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20260925 03:04:28.238446 24619 pending_rounds.cc:76] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c: Aborting all ops after (but not including) 1
I20260925 03:04:28.238461 24586 leader_election.cc:302] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:28.238659 24619 pending_rounds.cc:97] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260925 03:04:28.239547 24619 raft_consensus.cc:2866] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260925 03:04:28.240900 24619 raft_consensus.cc:1226] T c88cbd39e01f44628a094af516be2833 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 3. Current term is 4. Ops: []
I20260925 03:04:28.241357 24586 leader_election.cc:302] T a4abe0ea757e4886be4945c5001e529a P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 1bc1f2e62ab44af995165bd5116a9d43, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:28.244784 24586 leader_election.cc:302] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:04:28.223407 25788 raft_consensus.cc:3037] T 7e49248643df4131a1c42e74244e05c2 P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:28.249691 24586 leader_election.cc:302] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:28.251083 24472 raft_consensus.cc:1226] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Rejecting Update request from peer 1bc1f2e62ab44af995165bd5116a9d43 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260925 03:04:28.254523 24586 leader_election.cc:302] T 420163a6cdce412ebc21c25fe3835638 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:04:28.259260 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
W20260925 03:04:28.259438 24585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.4:43979, user_credentials={real_user=slave}} blocked reactor thread for 296067us
I20260925 03:04:28.262801 24586 leader_election.cc:302] T 48ca6ccd98194cf2af7b5278d15bf4ea P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:04:28.222543 25790 raft_consensus.cc:504] T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:28.248387 25911 raft_consensus.cc:2781] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260925 03:04:28.245558 25964 raft_consensus.cc:2726] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260925 03:04:28.286356 24437 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.516s user 0.009s sys 0.020s
I20260925 03:04:28.286657 24615 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38570 (request call id 1173) took 2120 ms. Trace:
W20260925 03:04:28.286682 24437 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.516s user 0.009s sys 0.020s
W20260925 03:04:28.287921 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 518078us
I20260925 03:04:28.287091 24615 rpcz_store.cc:276] 0925 03:04:26.166115 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:27.335477 (+1169362us) service_pool.cc:227] Handling call
0925 03:04:27.804197 (+468720us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:27.811460 (+ 7263us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:28.286460 (+475000us) spinlock_profiling.cc:243] Waited 455 ms on lock 0x7b5400249d50. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764cfa09 00007f437b4112e9 00007f4374af7684 00007f4374af7612 00007f4374af7541 00007f4374af74bc 00007f4374af6772 00007f43748fa94c 00007f43748f9d86 00007f43748f972f
0925 03:04:28.286603 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1047309568}
I20260925 03:04:28.288664 24437 leader_election.cc:302] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:28.248023 25962 raft_consensus.cc:2726] T a4abe0ea757e4886be4945c5001e529a P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 03:04:28.269901 25786 raft_consensus.cc:482] T a4abe0ea757e4886be4945c5001e529a P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:28.306367 25968 raft_consensus.cc:2781] T 48ca6ccd98194cf2af7b5278d15bf4ea P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 03:04:28.306861 25968 raft_consensus.cc:482] T 48ca6ccd98194cf2af7b5278d15bf4ea P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting leader election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:28.307195 25968 raft_consensus.cc:3037] T 48ca6ccd98194cf2af7b5278d15bf4ea P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:28.307246 24622 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38564 (request call id 1163) took 1544 ms. Trace:
I20260925 03:04:28.307653 24622 rpcz_store.cc:276] 0925 03:04:26.762762 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:27.877098 (+1114336us) service_pool.cc:227] Handling call
0925 03:04:27.897580 (+ 20482us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:04:27.897888 (+ 308us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:27.897916 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:04:27.937019 (+ 39103us) log.cc:842] Serialized 26 byte log entry
0925 03:04:28.284240 (+347221us) raft_consensus.cc:1631] Marking committed up to 2
0925 03:04:28.302885 (+ 18645us) log.cc:842] Serialized 14 byte log entry
0925 03:04:28.305908 (+ 3023us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:28.305984 (+ 76us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:04:28.306027 (+ 43us) raft_consensus.cc:1671] finished
0925 03:04:28.307042 (+ 1015us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:28.307189 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":18466,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":47,"raft.queue_time_us":12897,"raft.run_cpu_time_us":221,"raft.run_wall_time_us":9185,"spinlock_wait_cycles":13056,"thread_start_us":311177,"threads_started":2,"wal-append.queue_time_us":341472}
I20260925 03:04:28.265959 25961 raft_consensus.cc:2726] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260925 03:04:28.321957 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 58058.9us
I20260925 03:04:28.324541 24586 leader_election.cc:302] T 100c113a9fd84cc8adf2ec7b33ea0093 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5
W20260925 03:04:28.324551 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 195929us
I20260925 03:04:28.321434 24437 leader_election.cc:302] T 8e5819c15eac4544a57e5ee6c9d4f117 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:04:28.281435 25911 raft_consensus.cc:482] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:28.337339 24437 leader_election.cc:302] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:28.328527 24621 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38564 (request call id 1158) took 1567 ms. Trace:
I20260925 03:04:28.333225 24586 leader_election.cc:302] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:28.337962 24621 rpcz_store.cc:276] 0925 03:04:26.761465 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:27.861002 (+1099537us) service_pool.cc:227] Handling call
0925 03:04:27.861289 (+ 287us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:27.861481 (+ 192us) raft_consensus.cc:1520] Early marking committed up to index 2
0925 03:04:28.232057 (+370576us) log.cc:842] Serialized 14 byte log entry
0925 03:04:28.328151 (+ 96094us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:28.328207 (+ 56us) raft_consensus.cc:1631] Marking committed up to 2
0925 03:04:28.328250 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:28.328323 (+ 73us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:28.328461 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":168415,"prepare.run_cpu_time_us":71,"prepare.run_wall_time_us":2197,"thread_start_us":440678,"threads_started":3,"wal-append.queue_time_us":70814}
I20260925 03:04:28.340229 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c88cbd39e01f44628a094af516be2833" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:28.341573 24586 leader_election.cc:302] T 37a157996e944db2906110d87ecb0c54 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
W20260925 03:04:28.343474 25165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 84445.7us
W20260925 03:04:28.343993 25901 log.cc:924] Time spent T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5: Append to log took a long time: real 1.099s user 0.003s sys 0.002s
I20260925 03:04:28.346428 24437 leader_election.cc:302] T 293ef69caf514ac98510a68495f54eda P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 1bc1f2e62ab44af995165bd5116a9d43
I20260925 03:04:28.347380 24437 leader_election.cc:302] T b4ebd2b043024a548005c8b6b00b0449 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: 1bc1f2e62ab44af995165bd5116a9d43, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:28.348450 24437 leader_election.cc:302] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:04:28.334118 25751 raft_consensus.cc:2726] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260925 03:04:28.352211 25958 raft_consensus.cc:2726] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260925 03:04:28.338366 25911 raft_consensus.cc:3037] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:28.337177 25916 leader_election.cc:288] T c88cbd39e01f44628a094af516be2833 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:28.350816 24437 leader_election.cc:302] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:04:28.394222 25832 raft_consensus.cc:2781] T 8e5819c15eac4544a57e5ee6c9d4f117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 03:04:28.394049 25965 consensus_queue.cc:1035] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.074s
I20260925 03:04:28.394641 25832 raft_consensus.cc:482] T 8e5819c15eac4544a57e5ee6c9d4f117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:28.394829 25741 raft_consensus.cc:2781] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 03:04:28.396189 24617 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38564 (request call id 1157) took 1635 ms. Trace:
I20260925 03:04:28.397850 24617 rpcz_store.cc:276] 0925 03:04:26.761208 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:27.847073 (+1085865us) service_pool.cc:227] Handling call
0925 03:04:28.395511 (+548438us) spinlock_profiling.cc:243] Waited 993 us on lock 0x7b58001d3d98. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f43733ecf11 00007f4373570851 00007f4373570702 00007f437c460b09 00007f437b39c730 00007f437b46279b 00007f437b462821 00007f437b4262b0 00007f437b4117f9 00007f437b4112a2 00007f4374af7684
0925 03:04:28.395721 (+ 210us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:28.395891 (+ 170us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:28.395918 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:28.395955 (+ 37us) raft_consensus.cc:1631] Marking committed up to 1
0925 03:04:28.395980 (+ 25us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:28.396032 (+ 52us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:28.396134 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2282880}
W20260925 03:04:28.352778 25864 log.cc:924] Time spent T 11bb46bce3564841b7674e6a693bd2e6 P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 2.022s user 0.001s sys 0.005s
I20260925 03:04:28.400427 24437 leader_election.cc:302] T 06f4a7eaf6534e4986c9fd35911038e4 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:28.279946 25960 consensus_queue.cc:1035] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.119s
W20260925 03:04:28.393047 25885 consensus_meta.cc:287] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c: Time spent flushing consensus metadata: real 0.664s user 0.008s sys 0.006s
I20260925 03:04:28.322808 25786 raft_consensus.cc:504] T a4abe0ea757e4886be4945c5001e529a P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:28.395464 25741 raft_consensus.cc:686] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 LEADER]: Becoming Leader. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:04:28.398335 25962 raft_consensus.cc:2781] T 100c113a9fd84cc8adf2ec7b33ea0093 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260925 03:04:28.404983 25941 log.cc:924] Time spent T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.233s user 0.001s sys 0.000s
I20260925 03:04:28.406275 24620 raft_consensus.cc:2454] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 3.
I20260925 03:04:28.395824 24465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c85f5d1d3f004cf5817bb594fd8e6469" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:28.399025 25832 raft_consensus.cc:3037] T 8e5819c15eac4544a57e5ee6c9d4f117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:28.353053 25698 raft_consensus.cc:2781] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260925 03:04:28.416118 25865 consensus_queue.cc:1035] T c85f5d1d3f004cf5817bb594fd8e6469 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.096s
I20260925 03:04:28.316514 25968 raft_consensus.cc:504] T 48ca6ccd98194cf2af7b5278d15bf4ea P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:28.361845 25929 raft_consensus.cc:2726] T b4ebd2b043024a548005c8b6b00b0449 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260925 03:04:28.360908 25957 raft_consensus.cc:2781] T 293ef69caf514ac98510a68495f54eda P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260925 03:04:28.400401 25961 raft_consensus.cc:2726] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260925 03:04:28.403249 25845 raft_consensus.cc:2726] T 06f4a7eaf6534e4986c9fd35911038e4 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260925 03:04:28.425376 25962 raft_consensus.cc:482] T 100c113a9fd84cc8adf2ec7b33ea0093 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting leader election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
W20260925 03:04:28.446663 25918 consensus_meta.cc:287] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c: Time spent flushing consensus metadata: real 0.770s user 0.008s sys 0.004s
I20260925 03:04:28.410969 25936 leader_election.cc:288] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:28.446630 25962 raft_consensus.cc:3037] T 100c113a9fd84cc8adf2ec7b33ea0093 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:28.407424 25761 raft_consensus.cc:504] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:28.443423 25968 leader_election.cc:288] T 48ca6ccd98194cf2af7b5278d15bf4ea P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:28.430617 25698 raft_consensus.cc:482] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:28.444168 25957 raft_consensus.cc:482] T 293ef69caf514ac98510a68495f54eda P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1bc1f2e62ab44af995165bd5116a9d43)
I20260925 03:04:28.386811 25788 raft_consensus.cc:504] T 7e49248643df4131a1c42e74244e05c2 P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:28.426137 25855 raft_consensus.cc:482] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:28.361595 25955 consensus_queue.cc:1035] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.105s
I20260925 03:04:28.412518 25885 raft_consensus.cc:504] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:28.455518 25698 raft_consensus.cc:3037] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:28.471081 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9c33fd100ff44c1a3247319a324a044" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:28.457238 25957 raft_consensus.cc:3037] T 293ef69caf514ac98510a68495f54eda P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:28.455824 25790 leader_election.cc:288] T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:28.447080 25918 raft_consensus.cc:504] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:28.480064 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35992316cb8d4c028333388eff02ecde" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:28.394065 25963 consensus_queue.cc:1046] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.178s
I20260925 03:04:28.484030 24614 raft_consensus.cc:1226] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Rejecting Update request from peer 1bc1f2e62ab44af995165bd5116a9d43 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260925 03:04:28.485989 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5278a60f97da4e10aecd50a8d9a9a43c" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:28.489526 25918 leader_election.cc:288] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:28.495014 24615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e54ca1757ee4e42956a9cebbeca4f62" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:28.460667 25855 raft_consensus.cc:504] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:28.458900 25922 raft_consensus.cc:504] T 0b5df303cd6a4d59aca5c18bbbff24ed P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:28.478966 25761 leader_election.cc:288] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:28.507478 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5332a5d329493991dff994e70ea92c" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
W20260925 03:04:28.493016 25923 consensus_meta.cc:287] T 550118bc1a0641ce9803bdbf43f2069f P eeab93abfdb141a895002ec9f312296c: Time spent flushing consensus metadata: real 0.595s user 0.013s sys 0.000s
I20260925 03:04:28.516076 25951 raft_consensus.cc:504] T cc2df40db08649c9a75e90000b878559 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:28.516758 25947 raft_consensus.cc:482] T b627ecb4a5384ef392aa695e06599c39 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:28.516981 25923 raft_consensus.cc:504] T 550118bc1a0641ce9803bdbf43f2069f P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:28.517396 25947 raft_consensus.cc:504] T b627ecb4a5384ef392aa695e06599c39 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:28.520895 24618 raft_consensus.cc:1226] T 550118bc1a0641ce9803bdbf43f2069f P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 1. Current term is 2. Ops: []
I20260925 03:04:28.520438 25923 leader_election.cc:288] T 550118bc1a0641ce9803bdbf43f2069f P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:28.468019 25741 consensus_queue.cc:237] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:28.490847 25791 leader_election.cc:288] T c88cbd39e01f44628a094af516be2833 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
W20260925 03:04:28.526569 24547 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 1155) took 2439 ms (client timeout 2164 ms). Trace:
W20260925 03:04:28.527031 24547 rpcz_store.cc:269] 0925 03:04:26.087392 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:26.172262 (+ 84870us) service_pool.cc:227] Handling call
0925 03:04:28.526319 (+2354057us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b540012d754. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d42ae 00007f437b411c27 00007f4374af9c04 00007f4374af9b92 00007f4374af9ac1 00007f4374af9a3c 00007f4374af8cf2 00007f43748fa94c 00007f43748f9d86 00007f43748f972f
0925 03:04:28.526487 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2428521344}
I20260925 03:04:28.521659 25977 raft_consensus.cc:2726] T 37a157996e944db2906110d87ecb0c54 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260925 03:04:28.520954 25947 leader_election.cc:288] T b627ecb4a5384ef392aa695e06599c39 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:28.533320 24613 raft_consensus.cc:1226] T cc2df40db08649c9a75e90000b878559 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 1. Current term is 2. Ops: []
W20260925 03:04:28.534287 25798 consensus_meta.cc:287] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5: Time spent flushing consensus metadata: real 0.830s user 0.011s sys 0.000s
I20260925 03:04:28.534719 25798 raft_consensus.cc:504] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:28.535216 25794 leader_election.cc:288] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:28.537025 24471 raft_consensus.cc:1226] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 3. Current term is 4. Ops: []
I20260925 03:04:28.533354 25786 leader_election.cc:288] T a4abe0ea757e4886be4945c5001e529a P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:28.542366 25197 raft_consensus.cc:2454] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 4.
I20260925 03:04:28.546545 25953 raft_consensus.cc:482] T 11bb46bce3564841b7674e6a693bd2e6 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:28.548174 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7691c6b475428c82d6181d693d26f3" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:28.549923 24622 raft_consensus.cc:2379] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 in current term 4: Already voted for candidate eeab93abfdb141a895002ec9f312296c in this term.
I20260925 03:04:28.548295 25953 raft_consensus.cc:504] T 11bb46bce3564841b7674e6a693bd2e6 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:28.549793 25911 raft_consensus.cc:504] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:28.531661 25962 raft_consensus.cc:504] T 100c113a9fd84cc8adf2ec7b33ea0093 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:28.569291 25962 leader_election.cc:288] T 100c113a9fd84cc8adf2ec7b33ea0093 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:28.578141 25911 leader_election.cc:288] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:28.565033 25953 leader_election.cc:288] T 11bb46bce3564841b7674e6a693bd2e6 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:28.583542 24621 raft_consensus.cc:1226] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260925 03:04:28.590826 25798 leader_election.cc:288] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 election: Requested vote from peers eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:28.592918 24543 raft_consensus.cc:1226] T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 1. Current term is 2. Ops: []
W20260925 03:04:28.596617 25976 log.cc:924] Time spent T 420163a6cdce412ebc21c25fe3835638 P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.084s user 0.000s sys 0.001s
I20260925 03:04:28.596457 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74443a0a1dc34d88a2f42e3b722db405" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:28.597330 24621 raft_consensus.cc:3037] T 74443a0a1dc34d88a2f42e3b722db405 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Advancing to term 4
W20260925 03:04:28.598827 24540 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33068 (request call id 68) took 1906 ms (client timeout 2017 ms). Trace:
W20260925 03:04:28.599635 24540 rpcz_store.cc:269] 0925 03:04:26.692036 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:26.695431 (+ 3395us) service_pool.cc:227] Handling call
0925 03:04:28.591317 (+1895886us) spinlock_profiling.cc:243] Waited 883 ms on lock 0x7b54003b6854. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d42ae 00007f437b411c27 00007f4374af9c04 00007f4374af9b92 00007f4374af9ac1 00007f4374af9a3c 00007f4374af8cf2 00007f43748fa94c 00007f43748f9d86 00007f43748f972f
0925 03:04:28.598699 (+ 7382us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2030636800}
I20260925 03:04:28.601292 25788 leader_election.cc:288] T 7e49248643df4131a1c42e74244e05c2 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 4 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:28.603837 25951 leader_election.cc:288] T cc2df40db08649c9a75e90000b878559 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:28.604354 24621 raft_consensus.cc:2454] T 74443a0a1dc34d88a2f42e3b722db405 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 4.
I20260925 03:04:28.605870 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c88cbd39e01f44628a094af516be2833" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:28.606630 24621 raft_consensus.cc:2379] T c88cbd39e01f44628a094af516be2833 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 in current term 4: Already voted for candidate eeab93abfdb141a895002ec9f312296c in this term.
I20260925 03:04:28.607976 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b627ecb4a5384ef392aa695e06599c39" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
W20260925 03:04:28.610378 24620 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38564 (request call id 1140) took 2611 ms (client timeout 1634 ms). Trace:
W20260925 03:04:28.610823 24620 rpcz_store.cc:269] 0925 03:04:25.999232 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:25.999369 (+ 137us) service_pool.cc:227] Handling call
0925 03:04:28.610207 (+2610838us) spinlock_profiling.cc:243] Waited 2.39 s on lock 0x7b54002193d4. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d42ae 00007f437b411c27 00007f4374af9c04 00007f4374af9b92 00007f4374af9ac1 00007f4374af9a3c 00007f4374af8cf2 00007f43748fa94c 00007f43748f9d86 00007f43748f972f
0925 03:04:28.610295 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5485633792}
I20260925 03:04:28.610402 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5aec38e66a947c681e6c2a72b05e6ca" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
W20260925 03:04:28.629618 24437 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.22.193.2:36389 timed out after 2.164s (SENT) [suppressed 20 similar messages]
I20260925 03:04:28.638518 24544 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 1157) took 2550 ms. Trace:
I20260925 03:04:28.638986 24544 rpcz_store.cc:276] 0925 03:04:26.087885 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:26.647547 (+559662us) service_pool.cc:227] Handling call
0925 03:04:26.685341 (+ 37794us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:26.685570 (+ 229us) raft_consensus.cc:1520] Early marking committed up to index 4
0925 03:04:26.972802 (+287232us) log.cc:842] Serialized 14 byte log entry
0925 03:04:27.350368 (+377566us) log.cc:842] Serialized 14 byte log entry
0925 03:04:28.637979 (+1287611us) spinlock_profiling.cc:243] Waited 4.4 ms on lock 0x7b180025cd20. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f437c4478b1 00007f437c454661 00007f437351f4ca 00007f437cc00f99 00007f43733e3990 00007f43735a5864 00007f43735a5155 00007f43764c83d0 00007f43764da576 00007f43764eb0d7 00007f43764eb058
0925 03:04:28.638150 (+ 171us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:28.638202 (+ 52us) raft_consensus.cc:1631] Marking committed up to 4
0925 03:04:28.638248 (+ 46us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:28.638326 (+ 78us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:28.638449 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":80608,"prepare.queue_time_us":92532,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":43,"raft.queue_time_us":34459,"raft.run_cpu_time_us":153,"raft.run_wall_time_us":31387,"spinlock_wait_cycles":10128128,"thread_start_us":286612,"threads_started":1,"wal-append.queue_time_us":543344}
W20260925 03:04:28.642911 24511 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.4:43979, user_credentials={real_user=slave}} blocked reactor thread for 1.12475e+06us
W20260925 03:04:28.645289 24546 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 1154) took 2558 ms (client timeout 2617 ms). Trace:
W20260925 03:04:28.645768 24546 rpcz_store.cc:269] 0925 03:04:26.087155 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:26.639896 (+552741us) service_pool.cc:227] Handling call
0925 03:04:28.637400 (+1997504us) spinlock_profiling.cc:243] Waited 1.35 s on lock 0x7b540034b7d4. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d42ae 00007f437b411c27 00007f4374af9c04 00007f4374af9b92 00007f4374af9ac1 00007f4374af9a3c 00007f4374af8cf2 00007f43748fa94c 00007f43748f9d86 00007f43748f972f
0925 03:04:28.645210 (+ 7810us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3115513216}
I20260925 03:04:28.651158 25922 leader_election.cc:288] T 0b5df303cd6a4d59aca5c18bbbff24ed P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:28.652283 24618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bb46bce3564841b7674e6a693bd2e6" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:28.653568 25649 consensus_queue.cc:1046] T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20260925 03:04:28.657624 25925 consensus_meta.cc:287] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c: Time spent flushing consensus metadata: real 0.775s user 0.007s sys 0.006s
I20260925 03:04:28.658775 24542 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 579) took 1422 ms. Trace:
I20260925 03:04:28.659180 24542 rpcz_store.cc:276] 0925 03:04:27.236671 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:27.255172 (+ 18501us) service_pool.cc:227] Handling call
0925 03:04:27.255478 (+ 306us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:27.255684 (+ 206us) raft_consensus.cc:1520] Early marking committed up to index 2
0925 03:04:27.335905 (+ 80221us) log.cc:842] Serialized 14 byte log entry
0925 03:04:28.658404 (+1322499us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:28.658474 (+ 70us) raft_consensus.cc:1631] Marking committed up to 2
0925 03:04:28.658517 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:28.658601 (+ 84us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:28.658719 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1040571,"prepare.queue_time_us":94504,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":47,"thread_start_us":281538,"threads_started":1,"wal-append.queue_time_us":315438}
I20260925 03:04:28.663971 24543 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 553) took 3189 ms. Trace:
I20260925 03:04:28.664402 24543 rpcz_store.cc:276] 0925 03:04:25.474737 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:25.474907 (+ 170us) service_pool.cc:227] Handling call
0925 03:04:27.147060 (+1672153us) spinlock_profiling.cc:243] Waited 2.55 ms on lock 0x7b5800003c98. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f43733ecf11 00007f4373570851 00007f4373570702 00007f437c460b09 00007f437b39c730 00007f437b46279b 00007f437b462821 00007f437b4262b0 00007f437b4117f9 00007f437b4112a2 00007f4374af7684
0925 03:04:28.592711 (+1445651us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:28.594887 (+ 2176us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:28.663757 (+ 68870us) spinlock_profiling.cc:243] Waited 1.45 s on lock 0x7b54003b6850. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764cfa09 00007f437b4112e9 00007f4374af7684 00007f4374af7612 00007f4374af7541 00007f4374af74bc 00007f4374af6772 00007f43748fa94c 00007f43748f9d86 00007f43748f972f
0925 03:04:28.663903 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3330087168}
W20260925 03:04:28.665303 25933 consensus_meta.cc:287] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43: Time spent flushing consensus metadata: real 0.505s user 0.006s sys 0.005s
I20260925 03:04:28.665835 25196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e92f377a79354a6ebb4eec67303c13e3" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43"
I20260925 03:04:28.666589 25196 raft_consensus.cc:3037] T e92f377a79354a6ebb4eec67303c13e3 P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:28.669353 25795 raft_consensus.cc:2726] T 8e5819c15eac4544a57e5ee6c9d4f117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260925 03:04:28.669531 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 1.04396e+06us
I20260925 03:04:28.682091 25196 raft_consensus.cc:2454] T e92f377a79354a6ebb4eec67303c13e3 P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 2.
I20260925 03:04:28.688944 25788 raft_consensus.cc:482] T 8ac466793d094325ace224708b2158ed P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:28.689582 25788 raft_consensus.cc:504] T 8ac466793d094325ace224708b2158ed P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:28.696830 25788 leader_election.cc:288] T 8ac466793d094325ace224708b2158ed P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:28.714907 24367 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:57656 (request call id 81) took 5054 ms. Trace:
W20260925 03:04:28.710423 25163 leader_election.cc:334] T e92f377a79354a6ebb4eec67303c13e3 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389): Timed out: RequestConsensusVote RPC to 127.22.193.2:36389 timed out after 2.017s (SENT)
I20260925 03:04:28.698122 24368 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39368 (request call id 195) took 7869 ms. Trace:
I20260925 03:04:28.704032 24613 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38570 (request call id 1181) took 1068 ms. Trace:
I20260925 03:04:28.709442 24614 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34546 (request call id 51) took 2028 ms. Trace:
I20260925 03:04:28.720196 24614 rpcz_store.cc:276] 0925 03:04:26.680827 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:27.731313 (+1050486us) service_pool.cc:227] Handling call
0925 03:04:27.731630 (+ 317us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:04:28.485733 (+754103us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:28.709151 (+223418us) spinlock_profiling.cc:243] Waited 752 ms on lock 0x7b54003ac354. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d0f51 00007f43764cf99e 00007f437b4112e9 00007f4374af7684 00007f4374af7612 00007f4374af7541 00007f4374af74bc 00007f4374af6772 00007f43748fa94c 00007f43748f9d86
0925 03:04:28.709365 (+ 214us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1729876224}
I20260925 03:04:28.721395 25163 leader_election.cc:302] T e92f377a79354a6ebb4eec67303c13e3 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:28.702432 25855 leader_election.cc:288] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:28.718082 24367 rpcz_store.cc:276] 0925 03:04:23.660222 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:23.660346 (+ 124us) service_pool.cc:227] Handling call
0925 03:04:28.714833 (+5054487us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0925 03:04:26.269099 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0925 03:04:26.453547 (+184448us) write_op.cc:432] Acquiring schema lock in shared mode
0925 03:04:26.462399 (+ 8852us) write_op.cc:435] Acquired schema lock
0925 03:04:26.485075 (+ 22676us) tablet.cc:660] Decoding operations
0925 03:04:26.708127 (+223052us) write_op.cc:620] Acquiring the partition lock for write op
0925 03:04:26.720157 (+ 12030us) write_op.cc:641] Partition lock acquired for write op
0925 03:04:26.746771 (+ 26614us) tablet.cc:683] Acquiring locks for 3 operations
0925 03:04:26.773911 (+ 27140us) tablet.cc:699] Row locks acquired
0925 03:04:26.773930 (+ 19us) write_op.cc:260] PREPARE: finished
0925 03:04:26.774111 (+ 181us) write_op.cc:270] Start()
0925 03:04:26.774256 (+ 145us) write_op.cc:276] Timestamp: P: 1790305466774061 usec, L: 0
0925 03:04:26.774274 (+ 18us) op_driver.cc:331] REPLICATION: starting
0925 03:04:26.774947 (+ 673us) log.cc:842] Serialized 1601 byte log entry
0925 03:04:27.782824 (+1007877us) op_driver.cc:444] REPLICATION: finished
0925 03:04:28.054091 (+271267us) write_op.cc:301] APPLY: starting
0925 03:04:28.054211 (+ 120us) tablet.cc:1365] starting BulkCheckPresence
0925 03:04:28.054450 (+ 239us) tablet.cc:1368] finished BulkCheckPresence
0925 03:04:28.054667 (+ 217us) tablet.cc:1370] starting ApplyRowOperation cycle
0925 03:04:28.129538 (+ 74871us) tablet.cc:1381] finished ApplyRowOperation cycle
0925 03:04:28.146692 (+ 17154us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0925 03:04:28.146744 (+ 52us) write_op.cc:312] APPLY: finished
0925 03:04:28.152436 (+ 5692us) log.cc:842] Serialized 34 byte log entry
0925 03:04:28.152622 (+ 186us) write_op.cc:489] Releasing partition, row and schema locks
0925 03:04:28.152756 (+ 134us) write_op.cc:454] Released schema lock
0925 03:04:28.152798 (+ 42us) write_op.cc:341] FINISH: Updating metrics
0925 03:04:28.153071 (+ 273us) write_op.cc:489] Releasing partition, row and schema locks
0925 03:04:28.153097 (+ 26us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":331309,"reported_tablets":4,"child_traces":[["op",{"apply.queue_time_us":229201,"apply.run_cpu_time_us":3550,"apply.run_wall_time_us":99341,"num_ops":3,"prepare.queue_time_us":1164112,"prepare.run_cpu_time_us":30336,"prepare.run_wall_time_us":909070,"raft.queue_time_us":286961,"raft.run_cpu_time_us":12371,"raft.run_wall_time_us":175650,"replication_time_us":1004864,"thread_start_us":1479320,"threads_started":4,"wal-append.queue_time_us":569459}]]}
I20260925 03:04:28.724773 24613 rpcz_store.cc:276] 0925 03:04:27.635737 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:28.085300 (+449563us) service_pool.cc:227] Handling call
0925 03:04:28.141279 (+ 55979us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:28.533791 (+392512us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:28.703801 (+170010us) spinlock_profiling.cc:243] Waited 382 ms on lock 0x7b54000749d4. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d0f51 00007f43764cf99e 00007f437b4112e9 00007f4374af7684 00007f4374af7612 00007f4374af7541 00007f4374af74bc 00007f4374af6772 00007f43748fa94c 00007f43748f9d86
0925 03:04:28.703970 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":879406336}
W20260925 03:04:28.719231 25931 consensus_meta.cc:287] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c: Time spent flushing consensus metadata: real 0.540s user 0.011s sys 0.001s
I20260925 03:04:28.729907 25885 leader_election.cc:288] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
W20260925 03:04:28.731217 25926 consensus_meta.cc:287] T 442d4a29b1d743599f4b6b4871e26668 P eeab93abfdb141a895002ec9f312296c: Time spent flushing consensus metadata: real 0.841s user 0.009s sys 0.005s
I20260925 03:04:28.665791 25933 raft_consensus.cc:504] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:28.731969 25788 raft_consensus.cc:2726] T e92f377a79354a6ebb4eec67303c13e3 P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 03:04:28.722163 24368 rpcz_store.cc:276] 0925 03:04:20.828813 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:20.828954 (+ 141us) service_pool.cc:227] Handling call
0925 03:04:28.697989 (+7869035us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0925 03:04:23.043540 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0925 03:04:23.062047 (+ 18507us) write_op.cc:432] Acquiring schema lock in shared mode
0925 03:04:23.062088 (+ 41us) write_op.cc:435] Acquired schema lock
0925 03:04:23.062108 (+ 20us) tablet.cc:660] Decoding operations
0925 03:04:23.092449 (+ 30341us) write_op.cc:620] Acquiring the partition lock for write op
0925 03:04:23.094326 (+ 1877us) write_op.cc:641] Partition lock acquired for write op
0925 03:04:23.095990 (+ 1664us) tablet.cc:683] Acquiring locks for 2 operations
0925 03:04:23.205184 (+109194us) tablet.cc:699] Row locks acquired
0925 03:04:23.205214 (+ 30us) write_op.cc:260] PREPARE: finished
0925 03:04:23.232871 (+ 27657us) write_op.cc:270] Start()
0925 03:04:23.267582 (+ 34711us) write_op.cc:276] Timestamp: P: 1790305463227944 usec, L: 0
0925 03:04:23.302124 (+ 34542us) op_driver.cc:331] REPLICATION: starting
0925 03:04:23.412614 (+110490us) log.cc:842] Serialized 922 byte log entry
0925 03:04:24.234215 (+821601us) op_driver.cc:444] REPLICATION: finished
0925 03:04:25.465568 (+1231353us) spinlock_profiling.cc:243] Waited 5.18 ms on lock 0x7b44000232cc. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f43733ecf11 00007f43734baa95 00007f43734bb4b2 00007f4376163511 00007f437615ee1a 00007f437358a5fe 00007f437c3c1a39 00007f43735a78bb 00007f43735a5cbf 00007f43735a7e22 00007f4379ee0410
0925 03:04:26.395514 (+929946us) write_op.cc:301] APPLY: starting
0925 03:04:26.403447 (+ 7933us) tablet.cc:1365] starting BulkCheckPresence
0925 03:04:26.445943 (+ 42496us) tablet.cc:1368] finished BulkCheckPresence
0925 03:04:26.448827 (+ 2884us) tablet.cc:1370] starting ApplyRowOperation cycle
0925 03:04:26.670198 (+221371us) tablet.cc:1381] finished ApplyRowOperation cycle
0925 03:04:26.713651 (+ 43453us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0925 03:04:26.747485 (+ 33834us) write_op.cc:312] APPLY: finished
0925 03:04:26.779325 (+ 31840us) log.cc:842] Serialized 28 byte log entry
0925 03:04:26.779495 (+ 170us) write_op.cc:489] Releasing partition, row and schema locks
0925 03:04:26.779591 (+ 96us) write_op.cc:454] Released schema lock
0925 03:04:26.779695 (+ 104us) write_op.cc:341] FINISH: Updating metrics
0925 03:04:26.780028 (+ 333us) write_op.cc:489] Releasing partition, row and schema locks
0925 03:04:26.780056 (+ 28us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":197509,"reported_tablets":5,"child_traces":[["op",{"apply.queue_time_us":2084950,"apply.run_cpu_time_us":3272,"apply.run_wall_time_us":395309,"num_ops":2,"prepare.queue_time_us":1341349,"prepare.run_cpu_time_us":16372,"prepare.run_wall_time_us":576402,"raft.queue_time_us":480678,"raft.run_cpu_time_us":79428,"raft.run_wall_time_us":1997036,"replication_time_us":883215,"spinlock_wait_cycles":11903744,"thread_start_us":3601270,"threads_started":4,"wal-append.queue_time_us":245614,"wal-append.run_cpu_time_us":20391,"wal-append.run_wall_time_us":1329467}]]}
I20260925 03:04:28.735571 25926 raft_consensus.cc:504] T 442d4a29b1d743599f4b6b4871e26668 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:28.723106 24371 catalog_manager.cc:5495] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c reported cstate change: term changed from 1 to 3, leader changed from 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1) to eeab93abfdb141a895002ec9f312296c (127.22.193.3). New cstate: current_term: 3 leader_uuid: "eeab93abfdb141a895002ec9f312296c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: HEALTHY } } }
I20260925 03:04:28.738722 25926 leader_election.cc:288] T 442d4a29b1d743599f4b6b4871e26668 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:28.658279 25925 raft_consensus.cc:504] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:28.741871 24622 raft_consensus.cc:1226] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 2. Current term is 3. Ops: []
I20260925 03:04:28.739997 24371 catalog_manager.cc:5495] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c reported cstate change: term changed from 1 to 3, leader changed from 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1) to eeab93abfdb141a895002ec9f312296c (127.22.193.3). New cstate: current_term: 3 leader_uuid: "eeab93abfdb141a895002ec9f312296c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: HEALTHY } } }
I20260925 03:04:28.744046 25925 leader_election.cc:288] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 3 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:28.747617 25817 raft_consensus.cc:482] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:28.748535 25933 leader_election.cc:288] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 4 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
W20260925 03:04:28.751920 24585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.4:43979, user_credentials={real_user=slave}} blocked reactor thread for 141849us
I20260925 03:04:28.743966 24371 catalog_manager.cc:5495] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c reported cstate change: term changed from 1 to 2, leader changed from a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2) to eeab93abfdb141a895002ec9f312296c (127.22.193.3). New cstate: current_term: 2 leader_uuid: "eeab93abfdb141a895002ec9f312296c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: HEALTHY } } }
I20260925 03:04:28.752635 24585 leader_election.cc:302] T b1e85248d3b04b979dd080caeeb02fdb P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:04:28.754166 24371 catalog_manager.cc:5495] T 11bb46bce3564841b7674e6a693bd2e6 P eeab93abfdb141a895002ec9f312296c reported cstate change: term changed from 2 to 3, leader changed from a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2) to eeab93abfdb141a895002ec9f312296c (127.22.193.3). New cstate: current_term: 3 leader_uuid: "eeab93abfdb141a895002ec9f312296c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: HEALTHY } } }
I20260925 03:04:28.756037 24371 catalog_manager.cc:5495] T 0b5df303cd6a4d59aca5c18bbbff24ed P eeab93abfdb141a895002ec9f312296c reported cstate change: term changed from 1 to 2, leader changed from a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2) to eeab93abfdb141a895002ec9f312296c (127.22.193.3). New cstate: current_term: 2 leader_uuid: "eeab93abfdb141a895002ec9f312296c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: HEALTHY } } }
W20260925 03:04:28.757428 24511 connection.cc:232] RPC call timeout handler was delayed by -1.1556314909994398s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1829s for a response
W20260925 03:04:28.757859 24511 connection.cc:232] RPC call timeout handler was delayed by -1.1412314909994166s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1845s for a response
W20260925 03:04:28.758427 24511 connection.cc:232] RPC call timeout handler was delayed by -1.11243149099937s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1877s for a response
W20260925 03:04:28.758728 24511 connection.cc:232] RPC call timeout handler was delayed by -1.0854314909993263s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1907s for a response
W20260925 03:04:28.759078 24511 connection.cc:232] RPC call timeout handler was delayed by -1.0152314909992128s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1985s for a response
I20260925 03:04:28.748392 25817 raft_consensus.cc:504] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:28.776494 25960 consensus_queue.cc:1046] T ceab6b0fccd54e909a64f66fe81a9a1f P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: INVALID_TERM, Last received: 2.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260925 03:04:28.780633 25960 raft_consensus.cc:3032] T ceab6b0fccd54e909a64f66fe81a9a1f P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 LEADER]: Stepping down as leader of term 2
I20260925 03:04:28.781045 25960 raft_consensus.cc:729] T ceab6b0fccd54e909a64f66fe81a9a1f P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
W20260925 03:04:28.781745 24473 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.630s user 0.008s sys 0.070s
W20260925 03:04:28.782970 24473 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.631s user 0.008s sys 0.070s
I20260925 03:04:28.783622 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5278a60f97da4e10aecd50a8d9a9a43c" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:28.784549 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74443a0a1dc34d88a2f42e3b722db405" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:28.784941 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abd5b8dc1b7e49e3aa9af900d73ff0e7" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:28.785091 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25102ba08e4e49388f6176badcd10a25" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:28.786417 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b8e0d260a884ecdb20f79de46c36b19" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:28.787019 24548 raft_consensus.cc:2454] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 2.
I20260925 03:04:28.787902 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7691c6b475428c82d6181d693d26f3" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:28.788486 24548 raft_consensus.cc:2454] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 2.
I20260925 03:04:28.777439 25783 raft_consensus.cc:979] T 0be4e0aa984f4f2b81a58e382e434ad5 P 869b0a8575b54d08ae2817fad5dd42d5: : Instructing follower eeab93abfdb141a895002ec9f312296c to start an election
I20260925 03:04:28.791704 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:28.792567 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15861cd28e354598a3c6bfc4337c34bb" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:28.765969 25922 raft_consensus.cc:2781] T b1e85248d3b04b979dd080caeeb02fdb P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260925 03:04:28.792704 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 451173us
I20260925 03:04:28.794060 24614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c81a1b3b616492ab6c964f45af36077" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:28.795413 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:28.797339 24621 raft_consensus.cc:1226] T 100c113a9fd84cc8adf2ec7b33ea0093 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 1. Current term is 2. Ops: []
I20260925 03:04:28.807032 24545 raft_consensus.cc:2454] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 3.
I20260925 03:04:28.734472 25931 raft_consensus.cc:504] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:28.807883 24548 raft_consensus.cc:3037] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:28.777593 25649 consensus_queue.cc:1225] T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Peer 1bc1f2e62ab44af995165bd5116a9d43 log is divergent from this leader: its last log entry 2.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260925 03:04:28.827018 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ac466793d094325ace224708b2158ed" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:28.828985 24615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5aec38e66a947c681e6c2a72b05e6ca" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:28.829151 24586 leader_election.cc:302] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:04:28.830727 24614 raft_consensus.cc:1226] T 48ca6ccd98194cf2af7b5278d15bf4ea P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 1. Current term is 2. Ops: []
I20260925 03:04:28.809651 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5278a60f97da4e10aecd50a8d9a9a43c" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:28.827847 24620 raft_consensus.cc:2454] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 2.
I20260925 03:04:28.833000 24586 leader_election.cc:302] T 1414dcfe53f64a78ae413531eb8ec117 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:28.835108 24586 leader_election.cc:302] T 71f461a6e2ff434993b78a2d077f0abd P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 1bc1f2e62ab44af995165bd5116a9d43, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:28.810179 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e8af2641e124f9589c5ccfdac785f63" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:28.791184 25843 raft_consensus.cc:482] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:28.838964 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35992316cb8d4c028333388eff02ecde" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:28.842757 24586 leader_election.cc:302] T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:28.778673 25698 raft_consensus.cc:504] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:28.803784 25796 raft_consensus.cc:482] T b030d84c670b48d5bff337ccc0c77626 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:28.845806 24586 leader_election.cc:302] T 43366a0a2f9d4344bf4e37ffd290ef8a P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:28.831362 25842 raft_consensus.cc:1067] T 0be4e0aa984f4f2b81a58e382e434ad5 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 LEADER]: Signalling peer eeab93abfdb141a895002ec9f312296c to start an election
I20260925 03:04:28.848033 24586 leader_election.cc:302] T 7993f9c3e2a44f408bbf5d1e395e2306 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:28.849267 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5332a5d329493991dff994e70ea92c" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:28.849378 24621 raft_consensus.cc:1226] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 3. Current term is 4. Ops: []
I20260925 03:04:28.804122 25890 raft_consensus.cc:482] T 48ca6ccd98194cf2af7b5278d15bf4ea P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:28.803992 25560 raft_consensus.cc:482] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:28.837610 25973 raft_consensus.cc:2726] T 1414dcfe53f64a78ae413531eb8ec117 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 03:04:28.838528 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:28.877050 24436 leader_election.cc:302] T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:28.811774 25922 raft_consensus.cc:482] T b1e85248d3b04b979dd080caeeb02fdb P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting leader election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:28.863029 24586 leader_election.cc:302] T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:28.892019 24586 leader_election.cc:302] T 3d89396576884f7c8901dba54f783947 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:04:28.893134 24586 leader_election.cc:302] T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:04:28.894174 24586 leader_election.cc:302] T 21c43af5e6844fdaacc8c625fa01fd8d P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
W20260925 03:04:28.866093 24588 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.516s user 0.011s sys 0.030s
W20260925 03:04:28.895484 24588 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.545s user 0.012s sys 0.030s
I20260925 03:04:28.876052 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7691c6b475428c82d6181d693d26f3" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:28.895989 24546 raft_consensus.cc:3037] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Advancing to term 4
W20260925 03:04:28.853437 24470 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.689s user 0.011s sys 0.058s
W20260925 03:04:28.865899 25165 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.522s user 0.021s sys 0.028s
W20260925 03:04:28.896709 24470 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.732s user 0.011s sys 0.058s
I20260925 03:04:28.832144 25960 consensus_queue.cc:260] T ceab6b0fccd54e909a64f66fe81a9a1f P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:28.906522 24586 leader_election.cc:302] T 367277daf8d04778b3fa4ed94d71580d P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5
W20260925 03:04:28.897010 25165 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.553s user 0.021s sys 0.028s
I20260925 03:04:28.847350 25843 raft_consensus.cc:504] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:28.903621 24539 raft_consensus.cc:2454] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 3.
I20260925 03:04:28.897653 25560 raft_consensus.cc:504] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:28.901409 25765 raft_consensus.cc:2726] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260925 03:04:28.903615 25751 raft_consensus.cc:2781] T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 03:04:28.910386 25751 raft_consensus.cc:482] T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting leader election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:28.853776 25925 raft_consensus.cc:2726] T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260925 03:04:28.876646 25885 raft_consensus.cc:2726] T 7993f9c3e2a44f408bbf5d1e395e2306 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260925 03:04:28.907512 25165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 563378us
I20260925 03:04:28.910283 25957 raft_consensus.cc:504] T 293ef69caf514ac98510a68495f54eda P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:28.900758 25953 raft_consensus.cc:2781] T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader election won for term 4
I20260925 03:04:28.903455 25950 raft_consensus.cc:2781] T 3d89396576884f7c8901dba54f783947 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 03:04:28.860874 25926 raft_consensus.cc:2726] T 43366a0a2f9d4344bf4e37ffd290ef8a P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 03:04:28.899714 25922 raft_consensus.cc:3037] T b1e85248d3b04b979dd080caeeb02fdb P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:28.927024 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7691c6b475428c82d6181d693d26f3" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:28.908839 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74443a0a1dc34d88a2f42e3b722db405" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:28.900560 25942 raft_consensus.cc:2726] T 21c43af5e6844fdaacc8c625fa01fd8d P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 03:04:28.929478 24467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48ca6ccd98194cf2af7b5278d15bf4ea" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:28.944589 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ac285175464fbda67211d9574db415" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:28.929941 24548 raft_consensus.cc:2396] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 for term 4 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260925 03:04:28.936334 24472 raft_consensus.cc:2379] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c in current term 4: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:04:28.901199 25855 raft_consensus.cc:2726] T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260925 03:04:28.937572 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e725b82985746b39c0cc8f671c95353" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:28.945451 24467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "100c113a9fd84cc8adf2ec7b33ea0093" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:28.876152 25897 consensus_queue.cc:1046] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.047s
I20260925 03:04:28.933364 25793 raft_consensus.cc:482] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1bc1f2e62ab44af995165bd5116a9d43)
I20260925 03:04:28.940510 24615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e92f377a79354a6ebb4eec67303c13e3" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:28.921432 25973 raft_consensus.cc:2781] T 367277daf8d04778b3fa4ed94d71580d P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260925 03:04:28.864244 25796 raft_consensus.cc:504] T b030d84c670b48d5bff337ccc0c77626 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:28.910825 25751 raft_consensus.cc:3037] T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:28.955816 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc2df40db08649c9a75e90000b878559" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:28.924019 25165 leader_election.cc:302] T 29637f5a4a504ce6ba29033ccd0cd716 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:28.946722 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b5df303cd6a4d59aca5c18bbbff24ed" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:28.957021 24619 raft_consensus.cc:2354] T 0b5df303cd6a4d59aca5c18bbbff24ed P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 3. Current term is 4.
I20260925 03:04:28.939810 25698 leader_election.cc:288] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:28.947991 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c88cbd39e01f44628a094af516be2833" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:28.950148 24511 leader_election.cc:302] T cc2df40db08649c9a75e90000b878559 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:04:28.954551 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9c33fd100ff44c1a3247319a324a044" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:28.950225 24622 raft_consensus.cc:1226] T 74443a0a1dc34d88a2f42e3b722db405 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 3. Current term is 4. Ops: []
I20260925 03:04:28.959050 24511 leader_election.cc:302] T 550118bc1a0641ce9803bdbf43f2069f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:28.910778 25560 leader_election.cc:288] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:28.917922 24540 raft_consensus.cc:1261] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260925 03:04:28.960088 24511 leader_election.cc:302] T c85f5d1d3f004cf5817bb594fd8e6469 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:28.966734 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b627ecb4a5384ef392aa695e06599c39" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:28.972538 24511 leader_election.cc:302] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:28.943122 25950 raft_consensus.cc:482] T 3d89396576884f7c8901dba54f783947 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting leader election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:28.972546 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b5df303cd6a4d59aca5c18bbbff24ed" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:28.919319 25933 consensus_queue.cc:1046] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.039s
I20260925 03:04:28.914901 25783 consensus_queue.cc:1046] T 550118bc1a0641ce9803bdbf43f2069f P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.046s
I20260925 03:04:28.980854 25560 raft_consensus.cc:2726] T c85f5d1d3f004cf5817bb594fd8e6469 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260925 03:04:28.996505 24511 leader_election.cc:302] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
W20260925 03:04:28.994539 24616 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.595s user 0.014s sys 0.033s
I20260925 03:04:28.966568 25963 raft_consensus.cc:2726] T 29637f5a4a504ce6ba29033ccd0cd716 P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260925 03:04:29.023805 24616 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.626s user 0.014s sys 0.034s
I20260925 03:04:29.023902 25849 consensus_queue.cc:1046] T cc2df40db08649c9a75e90000b878559 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.009s
I20260925 03:04:28.950865 25649 consensus_queue.cc:1046] T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.025s
W20260925 03:04:29.002254 24436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 48255.8us
I20260925 03:04:28.957495 25973 raft_consensus.cc:482] T 367277daf8d04778b3fa4ed94d71580d P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting leader election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:29.005519 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5332a5d329493991dff994e70ea92c" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:28.967386 25828 raft_consensus.cc:2781] T cc2df40db08649c9a75e90000b878559 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 03:04:28.890026 25890 raft_consensus.cc:504] T 48ca6ccd98194cf2af7b5278d15bf4ea P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:28.963717 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "550118bc1a0641ce9803bdbf43f2069f" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:28.970829 25737 raft_consensus.cc:2781] T 550118bc1a0641ce9803bdbf43f2069f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 03:04:28.967124 25953 raft_consensus.cc:686] T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 LEADER]: Becoming Leader. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:04:28.995986 24467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5f9b503b0ab43999d582015a367441e" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:28.995498 25950 raft_consensus.cc:3037] T 3d89396576884f7c8901dba54f783947 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:28.999661 24464 raft_consensus.cc:3037] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:29.009701 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bb46bce3564841b7674e6a693bd2e6" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:28.993717 25805 raft_consensus.cc:2726] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260925 03:04:28.955334 25162 leader_election.cc:302] T 8ac466793d094325ace224708b2158ed P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:28.967208 25926 consensus_queue.cc:1046] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.043s
I20260925 03:04:28.970634 25799 consensus_queue.cc:1046] T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.043s
I20260925 03:04:28.960783 25793 raft_consensus.cc:504] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:29.002979 25197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e54ca1757ee4e42956a9cebbeca4f62" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43"
W20260925 03:04:29.071434 24617 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.565s user 0.016s sys 0.052s
I20260925 03:04:29.024658 25947 raft_consensus.cc:3032] T 550118bc1a0641ce9803bdbf43f2069f P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 LEADER]: Stepping down as leader of term 1
I20260925 03:04:29.082942 24472 raft_consensus.cc:2379] T 4e5332a5d329493991dff994e70ea92c P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c in current term 4: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:04:29.082871 24548 raft_consensus.cc:2454] T 11bb46bce3564841b7674e6a693bd2e6 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 3.
I20260925 03:04:29.046296 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442d4a29b1d743599f4b6b4871e26668" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:29.028795 24511 leader_election.cc:302] T 21c43af5e6844fdaacc8c625fa01fd8d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
W20260925 03:04:29.084239 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 414021us
W20260925 03:04:29.082908 24617 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.577s user 0.017s sys 0.052s
I20260925 03:04:29.012346 25817 leader_election.cc:288] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:29.024749 24616 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38570 (request call id 1182) took 1385 ms. Trace:
W20260925 03:04:29.058732 25832 consensus_meta.cc:287] T 8e5819c15eac4544a57e5ee6c9d4f117 P 869b0a8575b54d08ae2817fad5dd42d5: Time spent flushing consensus metadata: real 0.609s user 0.004s sys 0.009s
I20260925 03:04:29.126477 24546 raft_consensus.cc:2454] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 4.
I20260925 03:04:29.086206 25828 raft_consensus.cc:482] T cc2df40db08649c9a75e90000b878559 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:29.040753 25931 leader_election.cc:288] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:29.085165 25737 raft_consensus.cc:482] T 550118bc1a0641ce9803bdbf43f2069f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:29.129446 24464 raft_consensus.cc:2454] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 4.
W20260925 03:04:29.130158 24511 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 101477us
I20260925 03:04:29.130980 24511 leader_election.cc:302] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:29.116439 25890 leader_election.cc:288] T 48ca6ccd98194cf2af7b5278d15bf4ea P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:29.100944 25637 raft_consensus.cc:2781] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260925 03:04:29.028364 25964 raft_consensus.cc:2781] T 420163a6cdce412ebc21c25fe3835638 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260925 03:04:29.103701 24467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b030d84c670b48d5bff337ccc0c77626" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
W20260925 03:04:29.096843 25162 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 155481us
I20260925 03:04:29.113404 25804 raft_consensus.cc:3032] T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 LEADER]: Stepping down as leader of term 1
I20260925 03:04:29.114106 25793 leader_election.cc:288] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:29.114809 24470 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 623) took 1247 ms. Trace:
I20260925 03:04:29.117177 24622 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0be4e0aa984f4f2b81a58e382e434ad5"
dest_uuid: "eeab93abfdb141a895002ec9f312296c"
from {username='slave'} at 127.0.0.1:38570
I20260925 03:04:29.093837 25949 consensus_queue.cc:1046] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.029s
I20260925 03:04:29.094009 25865 consensus_queue.cc:1046] T 100c113a9fd84cc8adf2ec7b33ea0093 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.024s
I20260925 03:04:29.085198 25947 raft_consensus.cc:729] T 550118bc1a0641ce9803bdbf43f2069f P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:04:29.027022 25192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5aec38e66a947c681e6c2a72b05e6ca" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43"
I20260925 03:04:29.188865 24541 raft_consensus.cc:1226] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 2. Current term is 3. Ops: []
I20260925 03:04:29.190068 25893 raft_consensus.cc:3032] T cc2df40db08649c9a75e90000b878559 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 LEADER]: Stepping down as leader of term 1
I20260925 03:04:29.190014 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ac466793d094325ace224708b2158ed" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:29.190521 25893 raft_consensus.cc:729] T cc2df40db08649c9a75e90000b878559 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:04:29.190699 25516 raft_consensus.cc:3032] T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 LEADER]: Stepping down as leader of term 2
I20260925 03:04:29.191689 25516 raft_consensus.cc:729] T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:04:29.026101 25952 raft_consensus.cc:2726] T 71f461a6e2ff434993b78a2d077f0abd P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260925 03:04:29.192514 25516 consensus_queue.cc:260] T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:29.193558 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4abe0ea757e4886be4945c5001e529a" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:29.193821 25516 raft_consensus.cc:3037] T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:29.119230 25794 raft_consensus.cc:3032] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 LEADER]: Stepping down as leader of term 3
I20260925 03:04:29.194988 25794 raft_consensus.cc:729] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1bc1f2e62ab44af995165bd5116a9d43, State: Running, Role: LEADER
I20260925 03:04:29.195744 25794 consensus_queue.cc:260] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
W20260925 03:04:29.197084 25980 log.cc:924] Time spent T e6b6ec010caf4689b8c90449764f5cf9 P 1bc1f2e62ab44af995165bd5116a9d43: Append to log took a long time: real 0.194s user 0.001s sys 0.000s
I20260925 03:04:29.197165 25794 raft_consensus.cc:3037] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:29.199010 25195 raft_consensus.cc:1261] T 367277daf8d04778b3fa4ed94d71580d P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 LEARNER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260925 03:04:29.083362 25973 raft_consensus.cc:3037] T 367277daf8d04778b3fa4ed94d71580d P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:29.057730 24465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442d4a29b1d743599f4b6b4871e26668" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:29.043761 25560 raft_consensus.cc:2741] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260925 03:04:29.080312 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f68804eb5a244d87bb395d50683c6012" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:29.138114 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f1fac028a144058b92331c0d9b44ebf" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:29.139461 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06f4a7eaf6534e4986c9fd35911038e4" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:29.136943 24511 leader_election.cc:302] T 1414dcfe53f64a78ae413531eb8ec117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
W20260925 03:04:29.211737 24511 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 74870.3us
W20260925 03:04:29.074803 25636 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.676s user 0.026s sys 0.042s
I20260925 03:04:29.216940 24546 raft_consensus.cc:3037] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Advancing to term 4
W20260925 03:04:29.217059 25636 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.821s user 0.028s sys 0.042s
I20260925 03:04:29.141870 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c88cbd39e01f44628a094af516be2833" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:29.217701 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b5df303cd6a4d59aca5c18bbbff24ed" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:29.218317 24544 raft_consensus.cc:2362] T 0b5df303cd6a4d59aca5c18bbbff24ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 3. Re-sending same reply.
I20260925 03:04:29.218715 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7691c6b475428c82d6181d693d26f3" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:29.176093 24436 leader_election.cc:302] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5; no voters:
I20260925 03:04:29.177289 25796 leader_election.cc:288] T b030d84c670b48d5bff337ccc0c77626 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:29.144177 25851 raft_consensus.cc:482] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1bc1f2e62ab44af995165bd5116a9d43)
I20260925 03:04:29.219928 25851 raft_consensus.cc:504] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:29.177577 25947 consensus_queue.cc:260] T 550118bc1a0641ce9803bdbf43f2069f P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:29.222033 25851 leader_election.cc:288] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:29.222311 25947 raft_consensus.cc:3037] T 550118bc1a0641ce9803bdbf43f2069f P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:29.178154 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c85f5d1d3f004cf5817bb594fd8e6469" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:29.212507 24511 leader_election.cc:302] T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:04:29.223457 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e725b82985746b39c0cc8f671c95353" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:29.224047 24543 raft_consensus.cc:3037] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:29.182220 25865 raft_consensus.cc:3032] T 100c113a9fd84cc8adf2ec7b33ea0093 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 LEADER]: Stepping down as leader of term 1
I20260925 03:04:29.180558 25960 raft_consensus.cc:3037] T ceab6b0fccd54e909a64f66fe81a9a1f P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Advancing to term 3
W20260925 03:04:29.180545 24514 connection.cc:232] RPC call timeout handler was delayed by -1.5436694060008449s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21050000000000002s for a response
I20260925 03:04:29.229161 25865 raft_consensus.cc:729] T 100c113a9fd84cc8adf2ec7b33ea0093 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
W20260925 03:04:29.231510 24514 connection.cc:232] RPC call timeout handler was delayed by -1.5373694060017442s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21120000000000003s for a response
W20260925 03:04:29.231877 24514 connection.cc:232] RPC call timeout handler was delayed by -1.4590694060007081s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21989999999999998s for a response
I20260925 03:04:29.232234 25865 consensus_queue.cc:260] T 100c113a9fd84cc8adf2ec7b33ea0093 P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:04:29.233219 24514 connection.cc:232] RPC call timeout handler was delayed by -1.1702914920006151s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15560000000000002s for a response
W20260925 03:04:29.233592 24514 connection.cc:232] RPC call timeout handler was delayed by -1.1594914920005976s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15680000000000002s for a response
I20260925 03:04:29.175335 24470 rpcz_store.cc:276] 0925 03:04:27.867657 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:27.891899 (+ 24242us) service_pool.cc:227] Handling call
0925 03:04:27.913750 (+ 21851us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:04:28.154263 (+240513us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:28.154291 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:04:28.164005 (+ 9714us) log.cc:842] Serialized 26 byte log entry
0925 03:04:28.900303 (+736298us) raft_consensus.cc:1631] Marking committed up to 1
0925 03:04:28.900349 (+ 46us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:28.900417 (+ 68us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:04:29.114591 (+214174us) raft_consensus.cc:1671] finished
0925 03:04:29.114611 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:29.114741 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":88212,"raft.run_cpu_time_us":220,"raft.run_wall_time_us":220,"thread_start_us":975982,"threads_started":2,"wal-append.queue_time_us":903818}
I20260925 03:04:29.138123 25817 raft_consensus.cc:2781] T 21c43af5e6844fdaacc8c625fa01fd8d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 03:04:29.235491 25817 raft_consensus.cc:482] T 21c43af5e6844fdaacc8c625fa01fd8d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:29.101667 24616 rpcz_store.cc:276] 0925 03:04:27.638914 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:28.184162 (+545248us) service_pool.cc:227] Handling call
0925 03:04:28.219707 (+ 35545us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:28.236006 (+ 16299us) raft_consensus.cc:1520] Early marking committed up to index 2
0925 03:04:28.276760 (+ 40754us) log.cc:842] Serialized 14 byte log entry
0925 03:04:29.024388 (+747628us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:29.024444 (+ 56us) raft_consensus.cc:1631] Marking committed up to 2
0925 03:04:29.024485 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:29.024554 (+ 69us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:29.024676 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":15237,"prepare.queue_time_us":29503,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":44,"raft.queue_time_us":77813,"raft.run_cpu_time_us":216,"raft.run_wall_time_us":17047,"thread_start_us":689273,"threads_started":2,"wal-append.queue_time_us":185098}
I20260925 03:04:29.184330 24588 leader_election.cc:302] T 0b5df303cd6a4d59aca5c18bbbff24ed P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:29.176311 25804 raft_consensus.cc:729] T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:04:29.100852 25982 raft_consensus.cc:2781] T 8ac466793d094325ace224708b2158ed P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260925 03:04:29.132632 25828 raft_consensus.cc:3037] T cc2df40db08649c9a75e90000b878559 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:29.128315 25843 leader_election.cc:288] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:29.253609 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e54ca1757ee4e42956a9cebbeca4f62" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:29.227190 24465 raft_consensus.cc:3037] T 442d4a29b1d743599f4b6b4871e26668 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:29.134531 25737 raft_consensus.cc:3037] T 550118bc1a0641ce9803bdbf43f2069f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:29.191877 25191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e54ca1757ee4e42956a9cebbeca4f62" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43"
I20260925 03:04:29.213254 25560 raft_consensus.cc:2781] T 1414dcfe53f64a78ae413531eb8ec117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 03:04:29.202220 25953 consensus_queue.cc:237] T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:29.219132 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "550118bc1a0641ce9803bdbf43f2069f" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:29.229275 24511 leader_election.cc:302] T 100c113a9fd84cc8adf2ec7b33ea0093 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:04:29.184334 25698 raft_consensus.cc:3032] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 LEADER]: Stepping down as leader of term 2
I20260925 03:04:29.283118 25698 raft_consensus.cc:729] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:04:29.284162 25698 consensus_queue.cc:260] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:29.285553 25698 raft_consensus.cc:3037] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:29.191243 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48ca6ccd98194cf2af7b5278d15bf4ea" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:29.289021 24541 raft_consensus.cc:3037] T 48ca6ccd98194cf2af7b5278d15bf4ea P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:29.132409 25832 raft_consensus.cc:504] T 8e5819c15eac4544a57e5ee6c9d4f117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:04:29.239696 24436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.4:43979, user_credentials={real_user=slave}} blocked reactor thread for 63695.2us
I20260925 03:04:29.139956 25889 raft_consensus.cc:2741] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
W20260925 03:04:29.241516 25981 log.cc:924] Time spent T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5: Append to log took a long time: real 0.234s user 0.001s sys 0.000s
I20260925 03:04:29.249946 25979 raft_consensus.cc:2781] T 0b5df303cd6a4d59aca5c18bbbff24ed P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader election won for term 4
I20260925 03:04:29.252267 25190 raft_consensus.cc:1226] T e92f377a79354a6ebb4eec67303c13e3 P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 1. Current term is 2. Ops: []
W20260925 03:04:29.241397 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 57135.3us
I20260925 03:04:29.235881 25817 raft_consensus.cc:3037] T 21c43af5e6844fdaacc8c625fa01fd8d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:29.252863 25765 raft_consensus.cc:2781] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader election won for term 4
I20260925 03:04:29.125227 25957 leader_election.cc:288] T 293ef69caf514ac98510a68495f54eda P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:29.242396 24514 leader_election.cc:302] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:29.269132 24468 raft_consensus.cc:2379] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c in current term 4: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
W20260925 03:04:29.311736 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 69417.9us
I20260925 03:04:29.158882 25926 consensus_queue.cc:1046] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.080s
I20260925 03:04:29.271807 24546 raft_consensus.cc:2454] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 4.
I20260925 03:04:29.250108 25956 raft_consensus.cc:979] T 3d89396576884f7c8901dba54f783947 P 869b0a8575b54d08ae2817fad5dd42d5: : Instructing follower eeab93abfdb141a895002ec9f312296c to start an election
W20260925 03:04:29.280874 24511 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 51689.7us
I20260925 03:04:29.321491 24511 leader_election.cc:302] T 420163a6cdce412ebc21c25fe3835638 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:29.283195 25843 raft_consensus.cc:2781] T 100c113a9fd84cc8adf2ec7b33ea0093 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 03:04:29.134716 25741 consensus_queue.cc:1046] T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.078s
I20260925 03:04:29.297230 24436 leader_election.cc:302] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: 1bc1f2e62ab44af995165bd5116a9d43, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:29.252528 25637 raft_consensus.cc:482] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting leader election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:29.325350 24466 raft_consensus.cc:1261] T 06f4a7eaf6534e4986c9fd35911038e4 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260925 03:04:29.325721 25637 raft_consensus.cc:3037] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:29.328347 25984 raft_consensus.cc:3032] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [term 3 LEADER]: Stepping down as leader of term 3
I20260925 03:04:29.276970 25865 raft_consensus.cc:3037] T 100c113a9fd84cc8adf2ec7b33ea0093 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:29.302100 24542 raft_consensus.cc:2379] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c in current term 4: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:04:29.191339 25893 consensus_queue.cc:260] T cc2df40db08649c9a75e90000b878559 P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:29.333117 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc2df40db08649c9a75e90000b878559" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:29.334051 25979 raft_consensus.cc:686] T 0b5df303cd6a4d59aca5c18bbbff24ed P eeab93abfdb141a895002ec9f312296c [term 4 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:29.334466 25893 raft_consensus.cc:3037] T cc2df40db08649c9a75e90000b878559 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:29.334923 25979 consensus_queue.cc:237] T 0b5df303cd6a4d59aca5c18bbbff24ed P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:29.315181 24514 leader_election.cc:302] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:04:29.277323 25560 raft_consensus.cc:482] T 1414dcfe53f64a78ae413531eb8ec117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:29.319126 24471 raft_consensus.cc:1226] T 8e5819c15eac4544a57e5ee6c9d4f117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 1. Current term is 2. Ops: []
I20260925 03:04:29.274722 25982 raft_consensus.cc:482] T 8ac466793d094325ace224708b2158ed P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 FOLLOWER]: Starting leader election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:29.325042 25843 raft_consensus.cc:482] T 100c113a9fd84cc8adf2ec7b33ea0093 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:29.325770 24470 raft_consensus.cc:1261] T 9e8af2641e124f9589c5ccfdac785f63 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260925 03:04:29.345044 24511 leader_election.cc:302] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:29.329288 25938 raft_consensus.cc:979] T 1414dcfe53f64a78ae413531eb8ec117 P 869b0a8575b54d08ae2817fad5dd42d5: : Instructing follower eeab93abfdb141a895002ec9f312296c to start an election
W20260925 03:04:29.354413 24436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.4:43979, user_credentials={real_user=slave}} blocked reactor thread for 57258.8us
I20260925 03:04:29.356374 24436 leader_election.cc:302] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: 1bc1f2e62ab44af995165bd5116a9d43, eeab93abfdb141a895002ec9f312296c
W20260925 03:04:29.360453 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 45334.2us
I20260925 03:04:29.328810 25984 raft_consensus.cc:729] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:29.364656 24468 raft_consensus.cc:1226] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 3. Current term is 4. Ops: []
I20260925 03:04:29.329703 25964 raft_consensus.cc:482] T 420163a6cdce412ebc21c25fe3835638 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting leader election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:29.349427 25560 raft_consensus.cc:3037] T 1414dcfe53f64a78ae413531eb8ec117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:29.359292 25951 consensus_queue.cc:1035] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.021s
I20260925 03:04:29.386744 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c81a1b3b616492ab6c964f45af36077" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:29.351903 25849 raft_consensus.cc:1067] T 3d89396576884f7c8901dba54f783947 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 LEADER]: Signalling peer eeab93abfdb141a895002ec9f312296c to start an election
I20260925 03:04:29.348421 25843 raft_consensus.cc:3037] T 100c113a9fd84cc8adf2ec7b33ea0093 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:29.341099 25765 raft_consensus.cc:686] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 LEADER]: Becoming Leader. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:04:29.381847 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5aec38e66a947c681e6c2a72b05e6ca" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:29.383391 24467 raft_consensus.cc:1226] T 293ef69caf514ac98510a68495f54eda P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Rejecting Update request from peer 1bc1f2e62ab44af995165bd5116a9d43 for earlier term 2. Current term is 3. Ops: []
W20260925 03:04:29.365696 25988 log.cc:924] Time spent T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.127s user 0.000s sys 0.001s
I20260925 03:04:29.348059 25911 consensus_queue.cc:1035] T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.037s
I20260925 03:04:29.349040 25982 raft_consensus.cc:3037] T 8ac466793d094325ace224708b2158ed P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:29.395332 25897 raft_consensus.cc:1067] T 1414dcfe53f64a78ae413531eb8ec117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 LEADER]: Signalling peer eeab93abfdb141a895002ec9f312296c to start an election
I20260925 03:04:29.402516 25922 raft_consensus.cc:504] T b1e85248d3b04b979dd080caeeb02fdb P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:29.389504 25799 raft_consensus.cc:2781] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260925 03:04:29.379385 25819 consensus_queue.cc:1046] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.027s
I20260925 03:04:29.387981 25783 raft_consensus.cc:2726] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260925 03:04:29.369230 25934 raft_consensus.cc:2726] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260925 03:04:29.407935 25931 raft_consensus.cc:482] T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:29.352275 25952 raft_consensus.cc:3032] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [term 1 LEADER]: Stepping down as leader of term 1
I20260925 03:04:29.421978 24619 raft_consensus.cc:1261] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260925 03:04:29.423049 25165 leader_election.cc:302] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:29.427003 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:29.384371 25964 raft_consensus.cc:3037] T 420163a6cdce412ebc21c25fe3835638 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:29.372134 25926 consensus_queue.cc:1046] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.033s
I20260925 03:04:29.440443 25793 raft_consensus.cc:2781] T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260925 03:04:29.457340 24614 raft_consensus.cc:1226] T b1e85248d3b04b979dd080caeeb02fdb P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 1. Current term is 2. Ops: []
I20260925 03:04:29.468580 24543 raft_consensus.cc:2454] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 4.
I20260925 03:04:29.464675 25785 raft_consensus.cc:2726] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260925 03:04:29.460049 25165 leader_election.cc:302] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:29.361258 25804 consensus_queue.cc:260] T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:29.412811 25940 consensus_queue.cc:1035] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.042s
I20260925 03:04:29.439837 25842 raft_consensus.cc:482] T 7993f9c3e2a44f408bbf5d1e395e2306 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:29.444347 25799 raft_consensus.cc:482] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:29.471692 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b5df303cd6a4d59aca5c18bbbff24ed" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:29.477365 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1e85248d3b04b979dd080caeeb02fdb" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:29.478394 25909 raft_consensus.cc:2726] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260925 03:04:29.484581 24617 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38570 (request call id 1188) took 1016 ms. Trace:
I20260925 03:04:29.481268 24613 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3d89396576884f7c8901dba54f783947"
dest_uuid: "eeab93abfdb141a895002ec9f312296c"
from {username='slave'} at 127.0.0.1:38570
I20260925 03:04:29.482010 25737 raft_consensus.cc:504] T 550118bc1a0641ce9803bdbf43f2069f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:29.472242 25795 raft_consensus.cc:2726] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260925 03:04:29.449824 25698 consensus_queue.cc:1046] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.035s
I20260925 03:04:29.475772 25922 leader_election.cc:288] T b1e85248d3b04b979dd080caeeb02fdb P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:29.474146 24466 raft_consensus.cc:1226] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 3. Current term is 4. Ops: []
I20260925 03:04:29.488010 24543 raft_consensus.cc:3037] T 0b5df303cd6a4d59aca5c18bbbff24ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Advancing to term 4
W20260925 03:04:29.503772 24437 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.867s user 0.030s sys 0.063s
I20260925 03:04:29.489276 24617 rpcz_store.cc:276] 0925 03:04:28.467947 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:28.468315 (+ 368us) service_pool.cc:227] Handling call
0925 03:04:28.468667 (+ 352us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:04:28.469025 (+ 358us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:28.469049 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:04:28.476494 (+ 7445us) log.cc:842] Serialized 26 byte log entry
0925 03:04:29.095894 (+619400us) raft_consensus.cc:1631] Marking committed up to 1
0925 03:04:29.095934 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:29.096009 (+ 75us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:04:29.484375 (+388366us) raft_consensus.cc:1671] finished
0925 03:04:29.484395 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:29.484522 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2322,"raft.queue_time_us":1826,"raft.run_cpu_time_us":159,"raft.run_wall_time_us":5410,"thread_start_us":589834,"threads_started":1,"wal-append.queue_time_us":744105}
W20260925 03:04:29.504978 24437 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.870s user 0.030s sys 0.063s
W20260925 03:04:29.505265 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 875644us
I20260925 03:04:29.505604 24617 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1414dcfe53f64a78ae413531eb8ec117"
dest_uuid: "eeab93abfdb141a895002ec9f312296c"
from {username='slave'} at 127.0.0.1:38570
I20260925 03:04:29.506117 24617 raft_consensus.cc:482] T 1414dcfe53f64a78ae413531eb8ec117 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 03:04:29.506428 24617 raft_consensus.cc:3037] T 1414dcfe53f64a78ae413531eb8ec117 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:29.472445 25190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "293ef69caf514ac98510a68495f54eda" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43"
I20260925 03:04:29.491979 25799 raft_consensus.cc:3037] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:29.453665 25952 raft_consensus.cc:729] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:29.482077 24465 raft_consensus.cc:2454] T 442d4a29b1d743599f4b6b4871e26668 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 4.
I20260925 03:04:29.490648 25900 raft_consensus.cc:3032] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 LEADER]: Stepping down as leader of term 3
I20260925 03:04:29.492770 25165 leader_election.cc:302] T 7e49248643df4131a1c42e74244e05c2 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:29.491107 25842 raft_consensus.cc:504] T 7993f9c3e2a44f408bbf5d1e395e2306 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:29.438043 25984 consensus_queue.cc:260] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:29.515251 24544 raft_consensus.cc:2379] T 550118bc1a0641ce9803bdbf43f2069f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c in current term 2: Already voted for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in this term.
I20260925 03:04:29.497018 24616 raft_consensus.cc:1226] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260925 03:04:29.519990 24543 raft_consensus.cc:2454] T 0b5df303cd6a4d59aca5c18bbbff24ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 4.
I20260925 03:04:29.603461 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5f9b503b0ab43999d582015a367441e" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:29.604185 24543 raft_consensus.cc:2454] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 3.
I20260925 03:04:29.521301 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5332a5d329493991dff994e70ea92c" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:29.512063 25637 raft_consensus.cc:504] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:29.459520 25931 raft_consensus.cc:504] T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:29.522269 25900 raft_consensus.cc:729] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:04:29.501770 25832 leader_election.cc:288] T 8e5819c15eac4544a57e5ee6c9d4f117 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:29.557978 24588 leader_election.cc:302] T 442d4a29b1d743599f4b6b4871e26668 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:29.485836 25817 raft_consensus.cc:504] T 21c43af5e6844fdaacc8c625fa01fd8d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:29.560652 25842 leader_election.cc:288] T 7993f9c3e2a44f408bbf5d1e395e2306 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
W20260925 03:04:29.567237 25751 consensus_meta.cc:287] T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c: Time spent flushing consensus metadata: real 0.573s user 0.003s sys 0.007s
I20260925 03:04:29.495148 25933 consensus_queue.cc:1046] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.032s
I20260925 03:04:29.568507 25190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1e85248d3b04b979dd080caeeb02fdb" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43"
I20260925 03:04:29.529688 25804 raft_consensus.cc:3037] T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:29.524626 25909 raft_consensus.cc:2726] T 7e49248643df4131a1c42e74244e05c2 P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260925 03:04:29.588832 24586 leader_election.cc:302] T c88cbd39e01f44628a094af516be2833 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
W20260925 03:04:29.657904 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 69235.5us
I20260925 03:04:29.658689 24586 leader_election.cc:302] T c85f5d1d3f004cf5817bb594fd8e6469 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:29.589098 25799 raft_consensus.cc:504] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:29.503363 25765 consensus_queue.cc:237] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:29.593885 25973 raft_consensus.cc:504] T 367277daf8d04778b3fa4ed94d71580d P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } }
I20260925 03:04:29.514528 25849 consensus_queue.cc:1035] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.065s
I20260925 03:04:29.513190 25698 raft_consensus.cc:3032] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 LEADER]: Stepping down as leader of term 3
I20260925 03:04:29.672394 25698 raft_consensus.cc:729] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:04:29.673240 25698 consensus_queue.cc:260] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:29.674567 25698 raft_consensus.cc:3037] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:29.676031 25951 raft_consensus.cc:2726] T c85f5d1d3f004cf5817bb594fd8e6469 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260925 03:04:29.623594 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b030d84c670b48d5bff337ccc0c77626" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:29.677634 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4abe0ea757e4886be4945c5001e529a" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:29.677923 24585 leader_election.cc:302] T b1e85248d3b04b979dd080caeeb02fdb P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:04:29.680099 25951 raft_consensus.cc:2726] T b1e85248d3b04b979dd080caeeb02fdb P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260925 03:04:29.675959 25950 consensus_meta.cc:287] T 3d89396576884f7c8901dba54f783947 P eeab93abfdb141a895002ec9f312296c: Time spent flushing consensus metadata: real 0.583s user 0.007s sys 0.000s
I20260925 03:04:29.632905 25843 raft_consensus.cc:504] T 100c113a9fd84cc8adf2ec7b33ea0093 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:29.681020 25950 raft_consensus.cc:504] T 3d89396576884f7c8901dba54f783947 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:29.681358 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:29.682039 24545 raft_consensus.cc:3037] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:29.682171 24621 raft_consensus.cc:1226] T 3d89396576884f7c8901dba54f783947 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 1. Current term is 2. Ops: []
I20260925 03:04:29.682706 25843 leader_election.cc:288] T 100c113a9fd84cc8adf2ec7b33ea0093 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:29.682849 24613 raft_consensus.cc:482] T 3d89396576884f7c8901dba54f783947 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 03:04:29.683243 24613 raft_consensus.cc:3037] T 3d89396576884f7c8901dba54f783947 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:29.685209 25637 leader_election.cc:288] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:29.689273 25799 leader_election.cc:288] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:29.690227 25950 leader_election.cc:288] T 3d89396576884f7c8901dba54f783947 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:29.690393 25900 consensus_queue.cc:260] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:29.691430 25817 leader_election.cc:288] T 21c43af5e6844fdaacc8c625fa01fd8d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
W20260925 03:04:29.631652 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 102233us
W20260925 03:04:29.634285 24472 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.521s user 0.014s sys 0.026s
W20260925 03:04:29.725754 24472 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.613s user 0.014s sys 0.026s
I20260925 03:04:29.643743 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f68804eb5a244d87bb395d50683c6012" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:29.726397 24543 raft_consensus.cc:3037] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:29.639170 25191 raft_consensus.cc:2396] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260925 03:04:29.652709 24473 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 606) took 4043 ms. Trace:
I20260925 03:04:29.728253 24473 rpcz_store.cc:276] 0925 03:04:25.609480 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:26.653584 (+1044104us) service_pool.cc:227] Handling call
0925 03:04:28.018832 (+1365248us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:28.033389 (+ 14557us) raft_consensus.cc:1520] Early marking committed up to index 4
0925 03:04:28.151636 (+118247us) log.cc:842] Serialized 14 byte log entry
0925 03:04:29.021607 (+869971us) spinlock_profiling.cc:243] Waited 5 ms on lock 0x7b180015f9a0. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f437c4478b1 00007f437c454661 00007f437358a345 00007f437c3c1a39 00007f43735a78bb 00007f43735a5cbf 00007f43735a5155 00007f4379ea5167 00007f43764da807 00007f43764e9f0f 00007f43764e9e88
0925 03:04:29.446690 (+425083us) log.cc:842] Serialized 14 byte log entry
0925 03:04:29.449354 (+ 2664us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:29.449408 (+ 54us) raft_consensus.cc:1631] Marking committed up to 4
0925 03:04:29.449450 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:29.487126 (+ 37676us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:29.652490 (+165364us) spinlock_profiling.cc:243] Waited 1.35 s on lock 0x7b54001919d0. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764cfa09 00007f437b4112e9 00007f4374af7684 00007f4374af7612 00007f4374af7541 00007f4374af74bc 00007f4374af6772 00007f43748fa94c 00007f43748f9d86 00007f43748f972f
0925 03:04:29.652617 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10137,"prepare.queue_time_us":849696,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":42,"raft.queue_time_us":21802,"raft.run_cpu_time_us":285,"raft.run_wall_time_us":26408,"spinlock_wait_cycles":3106389504,"thread_start_us":1055483,"threads_started":2,"wal-append.queue_time_us":799489}
I20260925 03:04:29.653784 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b5df303cd6a4d59aca5c18bbbff24ed" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:29.730611 24464 raft_consensus.cc:2354] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 3. Current term is 4.
I20260925 03:04:29.730719 24614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e5819c15eac4544a57e5ee6c9d4f117" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:29.650969 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ac285175464fbda67211d9574db415" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:29.670392 24586 leader_election.cc:302] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
W20260925 03:04:29.733039 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 62750.4us
I20260925 03:04:29.733728 24586 leader_election.cc:302] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:04:29.638141 25911 raft_consensus.cc:2781] T 442d4a29b1d743599f4b6b4871e26668 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader election won for term 4
I20260925 03:04:29.691825 24620 raft_consensus.cc:1226] T 367277daf8d04778b3fa4ed94d71580d P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 3. Current term is 4. Ops: []
I20260925 03:04:29.692840 25973 leader_election.cc:288] T 367277daf8d04778b3fa4ed94d71580d P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:29.700088 25949 raft_consensus.cc:3032] T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 LEADER]: Stepping down as leader of term 1
I20260925 03:04:29.694334 25737 leader_election.cc:288] T 550118bc1a0641ce9803bdbf43f2069f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:29.699590 24541 raft_consensus.cc:2454] T 48ca6ccd98194cf2af7b5278d15bf4ea P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 2.
I20260925 03:04:29.700202 25978 raft_consensus.cc:2726] T c88cbd39e01f44628a094af516be2833 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260925 03:04:29.650091 25751 raft_consensus.cc:504] T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:29.758322 24616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7993f9c3e2a44f408bbf5d1e395e2306" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:29.758682 24619 raft_consensus.cc:1226] T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 1. Current term is 2. Ops: []
I20260925 03:04:29.759281 24622 raft_consensus.cc:482] T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
W20260925 03:04:29.701189 25938 consensus_queue.cc:1159] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "eeab93abfdb141a895002ec9f312296c" responder_term: 4 status { last_received { term: 1 index: 1 } last_committed_idx: 1 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 3. Current term is 4. Ops: [3.2-3.2]" } } last_received_current_leader { term: 0 index: 0 } }
I20260925 03:04:29.628129 24617 raft_consensus.cc:504] T 1414dcfe53f64a78ae413531eb8ec117 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:29.715715 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "100c113a9fd84cc8adf2ec7b33ea0093" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:29.762054 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c23337a7cb74184a0e57b7095cb6a11" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:29.633817 25828 raft_consensus.cc:504] T cc2df40db08649c9a75e90000b878559 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:29.649199 25947 raft_consensus.cc:482] T 37a157996e944db2906110d87ecb0c54 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:29.704422 25931 leader_election.cc:288] T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
W20260925 03:04:29.674685 25989 log.cc:924] Time spent T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.096s user 0.000s sys 0.000s
I20260925 03:04:29.744891 25925 raft_consensus.cc:2726] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260925 03:04:29.743072 24586 leader_election.cc:302] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:04:29.744118 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6b6ec010caf4689b8c90449764f5cf9" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
W20260925 03:04:29.746198 24547 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.680s user 0.028s sys 0.036s
I20260925 03:04:29.747112 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442d4a29b1d743599f4b6b4871e26668" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:29.691433 25900 raft_consensus.cc:3037] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:29.784708 24586 leader_election.cc:302] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
W20260925 03:04:29.784731 24547 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.719s user 0.028s sys 0.036s
I20260925 03:04:29.755373 24543 raft_consensus.cc:2454] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 3.
I20260925 03:04:29.744854 25783 consensus_queue.cc:1046] T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.012s
I20260925 03:04:29.757400 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367277daf8d04778b3fa4ed94d71580d" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:29.757639 25192 raft_consensus.cc:1226] T 7e49248643df4131a1c42e74244e05c2 P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 3. Current term is 4. Ops: []
I20260925 03:04:29.745462 25949 raft_consensus.cc:729] T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:04:29.759634 24622 raft_consensus.cc:3037] T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:29.763352 24614 raft_consensus.cc:1226] T 1414dcfe53f64a78ae413531eb8ec117 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 1. Current term is 2. Ops: []
W20260925 03:04:29.724756 24548 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.542s user 0.012s sys 0.020s
I20260925 03:04:29.764070 24617 leader_election.cc:288] T 1414dcfe53f64a78ae413531eb8ec117 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
W20260925 03:04:29.799788 24548 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.617s user 0.013s sys 0.020s
W20260925 03:04:29.773236 24511 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 423195us
I20260925 03:04:29.764813 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f68804eb5a244d87bb395d50683c6012" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:29.801219 24471 raft_consensus.cc:2454] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 3.
I20260925 03:04:29.689777 25964 raft_consensus.cc:504] T 420163a6cdce412ebc21c25fe3835638 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:29.623390 25952 consensus_queue.cc:260] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:29.778750 24546 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "74443a0a1dc34d88a2f42e3b722db405"
dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
mode: GRACEFUL
new_leader_uuid: "eeab93abfdb141a895002ec9f312296c"
from {username='slave'} at 127.0.0.1:40158
I20260925 03:04:29.759665 25751 leader_election.cc:288] T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:29.739590 24545 raft_consensus.cc:2454] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 4.
I20260925 03:04:29.692669 24613 raft_consensus.cc:504] T 3d89396576884f7c8901dba54f783947 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:29.746111 25911 raft_consensus.cc:686] T 442d4a29b1d743599f4b6b4871e26668 P eeab93abfdb141a895002ec9f312296c [term 4 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:29.785874 24542 raft_consensus.cc:2379] T cc2df40db08649c9a75e90000b878559 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c in current term 2: Already voted for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in this term.
I20260925 03:04:29.786828 24543 raft_consensus.cc:1261] T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260925 03:04:29.791047 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be4e0aa984f4f2b81a58e382e434ad5" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:29.764281 25973 raft_consensus.cc:2781] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader election won for term 4
I20260925 03:04:29.798198 25931 raft_consensus.cc:2781] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260925 03:04:29.802064 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9c33fd100ff44c1a3247319a324a044" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:29.804473 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d89396576884f7c8901dba54f783947" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:29.826862 24473 raft_consensus.cc:2379] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c in current term 4: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:04:29.803498 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15861cd28e354598a3c6bfc4337c34bb" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:29.800390 24548 raft_consensus.cc:2932] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:29.797770 24616 raft_consensus.cc:2379] T 7993f9c3e2a44f408bbf5d1e395e2306 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 in current term 2: Already voted for candidate eeab93abfdb141a895002ec9f312296c in this term.
I20260925 03:04:29.773970 25637 consensus_queue.cc:1046] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.025s
I20260925 03:04:29.816900 24545 raft_consensus.cc:1261] T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Refusing update from remote peer 1bc1f2e62ab44af995165bd5116a9d43: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260925 03:04:29.794972 25832 raft_consensus.cc:3032] T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 LEADER]: Stepping down as leader of term 1
I20260925 03:04:29.794921 24586 leader_election.cc:302] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:29.795385 25922 raft_consensus.cc:2726] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260925 03:04:29.802623 25190 raft_consensus.cc:1226] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 3. Current term is 4. Ops: []
W20260925 03:04:29.872471 25999 log.cc:924] Time spent T 04e068a0f80941dc9138cee063f53bab P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.071s user 0.000s sys 0.000s
I20260925 03:04:29.873823 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b030d84c670b48d5bff337ccc0c77626" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:29.874135 25560 raft_consensus.cc:504] T 1414dcfe53f64a78ae413531eb8ec117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:29.874864 24471 raft_consensus.cc:2396] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260925 03:04:29.875248 24618 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34546 (request call id 56) took 1080 ms. Trace:
I20260925 03:04:29.875674 24618 rpcz_store.cc:276] 0925 03:04:28.794739 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:28.803994 (+ 9255us) service_pool.cc:227] Handling call
0925 03:04:28.872304 (+ 68310us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:04:28.912030 (+ 39726us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:28.914812 (+ 2782us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:04:29.007679 (+ 92867us) log.cc:842] Serialized 26 byte log entry
0925 03:04:29.509103 (+501424us) raft_consensus.cc:1631] Marking committed up to 4
0925 03:04:29.520342 (+ 11239us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:29.528730 (+ 8388us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:04:29.875004 (+346274us) raft_consensus.cc:1671] finished
0925 03:04:29.875033 (+ 29us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:29.875189 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":439854,"threads_started":1,"wal-append.queue_time_us":733009}
I20260925 03:04:29.878786 25560 leader_election.cc:288] T 1414dcfe53f64a78ae413531eb8ec117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:29.854236 25804 consensus_queue.cc:1046] T 7993f9c3e2a44f408bbf5d1e395e2306 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.025s
W20260925 03:04:29.881943 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 89324.5us
I20260925 03:04:29.778059 25947 raft_consensus.cc:504] T 37a157996e944db2906110d87ecb0c54 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:29.882973 24586 leader_election.cc:302] T 550118bc1a0641ce9803bdbf43f2069f P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:29.861294 24511 leader_election.cc:302] T 43366a0a2f9d4344bf4e37ffd290ef8a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
W20260925 03:04:29.853935 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 492648us
I20260925 03:04:29.863973 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1414dcfe53f64a78ae413531eb8ec117" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:29.884835 24473 raft_consensus.cc:3032] T 1414dcfe53f64a78ae413531eb8ec117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 LEADER]: Stepping down as leader of term 1
I20260925 03:04:29.885175 24473 raft_consensus.cc:729] T 1414dcfe53f64a78ae413531eb8ec117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:04:29.886297 24511 leader_election.cc:302] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:04:29.889539 24622 raft_consensus.cc:504] T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:29.892334 24514 leader_election.cc:302] T a7aa47c48404465b94a36fd819fe3c7a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:29.893631 24514 leader_election.cc:302] T 3d89396576884f7c8901dba54f783947 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:29.896978 24586 leader_election.cc:302] T 100c113a9fd84cc8adf2ec7b33ea0093 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:29.898561 24586 leader_election.cc:302] T 48ca6ccd98194cf2af7b5278d15bf4ea P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:04:29.900197 24586 leader_election.cc:302] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:04:29.826347 25931 raft_consensus.cc:482] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:29.881121 25560 raft_consensus.cc:3032] T 7993f9c3e2a44f408bbf5d1e395e2306 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 LEADER]: Stepping down as leader of term 1
I20260925 03:04:29.854887 25973 raft_consensus.cc:686] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [term 4 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:29.859167 25832 raft_consensus.cc:729] T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:04:29.881220 25637 raft_consensus.cc:3032] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c [term 1 LEADER]: Stepping down as leader of term 1
I20260925 03:04:29.906565 25922 raft_consensus.cc:2726] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260925 03:04:29.907106 25804 raft_consensus.cc:2781] T 43366a0a2f9d4344bf4e37ffd290ef8a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 03:04:29.908841 25737 raft_consensus.cc:2726] T a7aa47c48404465b94a36fd819fe3c7a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 03:04:29.909068 25950 raft_consensus.cc:2726] T 100c113a9fd84cc8adf2ec7b33ea0093 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260925 03:04:29.882717 25783 consensus_queue.cc:1046] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.058s
I20260925 03:04:29.912850 25560 raft_consensus.cc:729] T 7993f9c3e2a44f408bbf5d1e395e2306 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:04:29.906417 25926 raft_consensus.cc:2726] T 550118bc1a0641ce9803bdbf43f2069f P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260925 03:04:29.936756 25979 raft_consensus.cc:2781] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader election won for term 3
I20260925 03:04:29.908950 25931 raft_consensus.cc:3037] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:29.917311 25983 raft_consensus.cc:2781] T 48ca6ccd98194cf2af7b5278d15bf4ea P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Leader election won for term 2
I20260925 03:04:29.905807 25851 raft_consensus.cc:2781] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 03:04:29.948027 25804 raft_consensus.cc:482] T 43366a0a2f9d4344bf4e37ffd290ef8a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:29.956139 25984 raft_consensus.cc:3037] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Advancing to term 4
W20260925 03:04:29.942415 25953 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.563s user 0.017s sys 0.019s
I20260925 03:04:29.913399 25949 consensus_queue.cc:260] T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:29.978502 25947 leader_election.cc:288] T 37a157996e944db2906110d87ecb0c54 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:29.943840 25911 consensus_queue.cc:237] T 442d4a29b1d743599f4b6b4871e26668 P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:29.974702 25973 consensus_queue.cc:237] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:29.973485 25983 raft_consensus.cc:686] T 48ca6ccd98194cf2af7b5278d15bf4ea P eeab93abfdb141a895002ec9f312296c [term 2 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:29.980792 25983 consensus_queue.cc:237] T 48ca6ccd98194cf2af7b5278d15bf4ea P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:04:29.985524 26001 log.cc:924] Time spent T 420163a6cdce412ebc21c25fe3835638 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.079s user 0.000s sys 0.001s
I20260925 03:04:29.969583 25979 raft_consensus.cc:686] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c [term 3 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
W20260925 03:04:29.994287 25982 consensus_meta.cc:287] T 8ac466793d094325ace224708b2158ed P 1bc1f2e62ab44af995165bd5116a9d43: Time spent flushing consensus metadata: real 0.549s user 0.005s sys 0.004s
I20260925 03:04:29.994799 25982 raft_consensus.cc:504] T 8ac466793d094325ace224708b2158ed P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:30.036303 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48ca6ccd98194cf2af7b5278d15bf4ea" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:30.041272 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:30.041723 24471 raft_consensus.cc:1226] T 293ef69caf514ac98510a68495f54eda P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Rejecting Update request from peer 1bc1f2e62ab44af995165bd5116a9d43 for earlier term 2. Current term is 3. Ops: []
I20260925 03:04:30.043896 24473 consensus_queue.cc:260] T 1414dcfe53f64a78ae413531eb8ec117 P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:29.915444 24543 raft_consensus.cc:3037] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:30.050338 25196 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48408 (request call id 123) took 1088 ms. Trace:
I20260925 03:04:29.893175 24541 raft_consensus.cc:1226] T f22ddd7d1c444dc58f45a1fb0b3e02bc P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260925 03:04:30.051004 25196 rpcz_store.cc:276] 0925 03:04:28.961681 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:28.967517 (+ 5836us) service_pool.cc:227] Handling call
0925 03:04:29.220416 (+252899us) spinlock_profiling.cc:243] Waited 13.9 ms on lock 0x7b5800004e98. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f43733ecf11 00007f4373570851 00007f4373570702 00007f437c460b09 00007f437b39c730 00007f437b46279b 00007f437b462821 00007f437b4262b0 00007f437b4117f9 00007f437b4112a2 00007f4374af7684
0925 03:04:29.220635 (+ 219us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:29.240290 (+ 19655us) raft_consensus.cc:1520] Early marking committed up to index 3
0925 03:04:29.808711 (+568421us) log.cc:842] Serialized 14 byte log entry
0925 03:04:30.049198 (+240487us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:30.049350 (+ 152us) raft_consensus.cc:1631] Marking committed up to 3
0925 03:04:30.049542 (+ 192us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:30.049699 (+ 157us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:30.050037 (+ 338us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":683405,"prepare.run_cpu_time_us":69,"prepare.run_wall_time_us":70,"spinlock_wait_cycles":31913856,"thread_start_us":630219,"threads_started":2,"wal-append.queue_time_us":158612}
W20260925 03:04:30.055927 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 127156us
I20260925 03:04:29.940678 25637 raft_consensus.cc:729] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:30.059134 25560 consensus_queue.cc:260] T 7993f9c3e2a44f408bbf5d1e395e2306 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:30.060776 25698 raft_consensus.cc:482] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:30.061611 25560 raft_consensus.cc:3037] T 7993f9c3e2a44f408bbf5d1e395e2306 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:30.061864 25698 raft_consensus.cc:504] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:30.059332 25637 consensus_queue.cc:260] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:30.065793 25698 leader_election.cc:288] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:30.065981 25637 raft_consensus.cc:3037] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:30.071970 24543 raft_consensus.cc:1261] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Refusing update from remote peer 1bc1f2e62ab44af995165bd5116a9d43: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260925 03:04:30.075316 25851 raft_consensus.cc:686] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 LEADER]: Becoming Leader. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:04:30.077317 25851 consensus_queue.cc:237] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:30.079286 25785 consensus_queue.cc:1046] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260925 03:04:30.079939 25819 consensus_queue.cc:1035] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260925 03:04:29.963537 25828 leader_election.cc:288] T cc2df40db08649c9a75e90000b878559 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:30.081234 25832 consensus_queue.cc:260] T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:29.971608 25804 raft_consensus.cc:3037] T 43366a0a2f9d4344bf4e37ffd290ef8a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:30.087991 24541 raft_consensus.cc:1261] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260925 03:04:30.089561 24540 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 1198) took 1140 ms. Trace:
W20260925 03:04:30.090395 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 196674us
I20260925 03:04:30.091663 25737 raft_consensus.cc:2726] T 3d89396576884f7c8901dba54f783947 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 03:04:30.083755 24616 raft_consensus.cc:1226] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 3. Current term is 4. Ops: []
W20260925 03:04:29.972529 25953 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.595s user 0.017s sys 0.019s
I20260925 03:04:30.147194 25805 raft_consensus.cc:2726] T 420163a6cdce412ebc21c25fe3835638 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260925 03:04:30.151912 24468 raft_consensus.cc:1261] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260925 03:04:30.152593 25636 raft_consensus.cc:3032] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c [term 3 LEADER]: Stepping down as leader of term 3
I20260925 03:04:30.153156 25636 raft_consensus.cc:729] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:30.129587 24540 rpcz_store.cc:276] 0925 03:04:28.949047 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:29.010470 (+ 61423us) service_pool.cc:227] Handling call
0925 03:04:29.065900 (+ 55430us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:04:29.143675 (+ 77775us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:29.159719 (+ 16044us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:04:29.279384 (+119665us) log.cc:842] Serialized 26 byte log entry
0925 03:04:29.798831 (+519447us) raft_consensus.cc:1631] Marking committed up to 2
0925 03:04:30.041404 (+242573us) log.cc:842] Serialized 14 byte log entry
0925 03:04:30.088813 (+ 47409us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:30.088987 (+ 174us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:04:30.089086 (+ 99us) raft_consensus.cc:1671] finished
0925 03:04:30.089105 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:30.089346 (+ 241us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":103476,"raft.queue_time_us":88766,"raft.run_cpu_time_us":264,"raft.run_wall_time_us":10425,"thread_start_us":665338,"threads_started":2,"wal-append.queue_time_us":570222}
I20260925 03:04:30.154767 25636 consensus_queue.cc:260] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:30.156385 25636 raft_consensus.cc:3037] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:30.162892 24471 raft_consensus.cc:1226] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 3. Current term is 4. Ops: []
I20260925 03:04:30.135087 24539 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 1196) took 1185 ms. Trace:
W20260925 03:04:30.165475 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 100642us
I20260925 03:04:30.169368 24541 raft_consensus.cc:1226] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 3. Current term is 4. Ops: []
I20260925 03:04:30.169194 25979 consensus_queue.cc:237] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:30.170795 25793 raft_consensus.cc:482] T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:30.171083 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37a157996e944db2906110d87ecb0c54" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:30.175110 25942 raft_consensus.cc:482] T 5278a60f97da4e10aecd50a8d9a9a43c P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:30.177974 25982 leader_election.cc:288] T 8ac466793d094325ace224708b2158ed P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 3 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:30.178903 24620 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38570 (request call id 1219) took 1040 ms. Trace:
I20260925 03:04:30.180630 25987 raft_consensus.cc:3032] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 LEADER]: Stepping down as leader of term 2
I20260925 03:04:30.214100 25987 raft_consensus.cc:729] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1bc1f2e62ab44af995165bd5116a9d43, State: Running, Role: LEADER
I20260925 03:04:30.182282 25984 raft_consensus.cc:482] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:30.182898 24547 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 1194) took 1266 ms. Trace:
W20260925 03:04:30.183010 24511 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 288988us
I20260925 03:04:30.183373 24614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60d45b7d467d4e1ab92367f6d7d52b15" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:30.187532 24546 raft_consensus.cc:595] T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 LEADER]: Received request to transfer leadership to TS eeab93abfdb141a895002ec9f312296c
I20260925 03:04:29.989586 25964 leader_election.cc:288] T 420163a6cdce412ebc21c25fe3835638 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:30.188925 24616 raft_consensus.cc:1226] T 420163a6cdce412ebc21c25fe3835638 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 3. Current term is 4. Ops: []
I20260925 03:04:30.178675 24473 raft_consensus.cc:3037] T 1414dcfe53f64a78ae413531eb8ec117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:30.192235 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c23337a7cb74184a0e57b7095cb6a11" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:30.192771 24622 leader_election.cc:288] T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 3 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:30.198570 24539 rpcz_store.cc:276] 0925 03:04:28.935197 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:29.000923 (+ 65726us) service_pool.cc:227] Handling call
0925 03:04:29.001204 (+ 281us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:04:29.017601 (+ 16397us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:29.017645 (+ 44us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:04:29.059640 (+ 41995us) log.cc:842] Serialized 26 byte log entry
0925 03:04:29.637659 (+578019us) raft_consensus.cc:1631] Marking committed up to 2
0925 03:04:30.043625 (+405966us) log.cc:842] Serialized 14 byte log entry
0925 03:04:30.092288 (+ 48663us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:30.092430 (+ 142us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:04:30.092507 (+ 77us) raft_consensus.cc:1671] finished
0925 03:04:30.092527 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:30.106147 (+ 13620us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":54066,"prepare.queue_time_us":502833,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":51,"raft.queue_time_us":117107,"raft.run_cpu_time_us":347,"raft.run_wall_time_us":11604,"thread_start_us":864178,"threads_started":2,"wal-append.queue_time_us":740637}
W20260925 03:04:30.195437 25165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 86024.3us
I20260925 03:04:30.206674 24620 rpcz_store.cc:276] 0925 03:04:29.138678 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:29.138882 (+ 204us) service_pool.cc:227] Handling call
0925 03:04:29.188233 (+ 49351us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:29.735872 (+547639us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:30.178396 (+442524us) spinlock_profiling.cc:243] Waited 503 ms on lock 0x7b54002ef4d4. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d0f51 00007f43764cf99e 00007f437b4112e9 00007f4374af7684 00007f4374af7612 00007f4374af7541 00007f4374af74bc 00007f4374af6772 00007f43748fa94c 00007f43748f9d86
0925 03:04:30.178710 (+ 314us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1157746560}
I20260925 03:04:30.208665 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "420163a6cdce412ebc21c25fe3835638" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:30.211251 24613 leader_election.cc:288] T 3d89396576884f7c8901dba54f783947 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 3 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:30.212492 25832 raft_consensus.cc:3037] T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:30.213505 25952 raft_consensus.cc:3037] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:30.210925 25949 raft_consensus.cc:3037] T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:30.220604 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367277daf8d04778b3fa4ed94d71580d" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:30.242544 24544 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 602) took 1053 ms. Trace:
I20260925 03:04:30.243335 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ac466793d094325ace224708b2158ed" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:30.249974 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be4e0aa984f4f2b81a58e382e434ad5" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:30.257834 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d89396576884f7c8901dba54f783947" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:30.258761 25805 consensus_queue.cc:1046] T c88cbd39e01f44628a094af516be2833 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260925 03:04:30.259341 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1414dcfe53f64a78ae413531eb8ec117" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:30.259533 24619 raft_consensus.cc:2454] T 37a157996e944db2906110d87ecb0c54 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 3.
I20260925 03:04:30.253307 24544 rpcz_store.cc:276] 0925 03:04:29.142337 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:29.785665 (+643328us) service_pool.cc:227] Handling call
0925 03:04:29.802023 (+ 16358us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:29.802203 (+ 180us) raft_consensus.cc:1520] Early marking committed up to index 2
0925 03:04:30.042250 (+240047us) log.cc:842] Serialized 14 byte log entry
0925 03:04:30.173678 (+131428us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:30.178178 (+ 4500us) raft_consensus.cc:1631] Marking committed up to 2
0925 03:04:30.179763 (+ 1585us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:30.181202 (+ 1439us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:30.195974 (+ 14772us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3022,"prepare.queue_time_us":361500,"prepare.run_cpu_time_us":66,"prepare.run_wall_time_us":70,"raft.queue_time_us":19336,"raft.run_cpu_time_us":119,"raft.run_wall_time_us":119,"thread_start_us":327004,"threads_started":2}
I20260925 03:04:30.260238 24541 raft_consensus.cc:2379] T 1414dcfe53f64a78ae413531eb8ec117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c in current term 2: Already voted for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in this term.
I20260925 03:04:30.263262 24542 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36786 (request call id 608) took 1026 ms. Trace:
I20260925 03:04:30.263643 24542 rpcz_store.cc:276] 0925 03:04:29.234303 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:29.332833 (+ 98530us) service_pool.cc:227] Handling call
0925 03:04:30.260893 (+928060us) spinlock_profiling.cc:243] Waited 452 ms on lock 0x7b54002823d4. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d42ae 00007f437b411c27 00007f4374af9c04 00007f4374af9b92 00007f4374af9ac1 00007f4374af9a3c 00007f4374af8cf2 00007f43748fa94c 00007f43748f9d86 00007f43748f972f
0925 03:04:30.261098 (+ 205us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1039349120}
I20260925 03:04:30.265342 25191 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49794 (request call id 132) took 1271 ms. Trace:
I20260925 03:04:30.265733 25191 rpcz_store.cc:276] 0925 03:04:28.994167 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:28.997172 (+ 3005us) service_pool.cc:227] Handling call
0925 03:04:29.191771 (+194599us) spinlock_profiling.cc:243] Waited 14.6 ms on lock 0x7b5800004e98. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f43733ecf11 00007f4373570851 00007f4373570bce 000000000034aea9 00007f437b39ca30 00007f437b4627b4 00007f437b462821 00007f437b4121b0 00007f437b411a94 00007f4374af9c04 00007f4374af9b92
0925 03:04:30.264871 (+1073100us) spinlock_profiling.cc:243] Waited 293 ms on lock 0x7b54006405d4. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d42ae 00007f437b411c27 00007f4374af9c04 00007f4374af9b92 00007f4374af9ac1 00007f4374af9a3c 00007f4374af8cf2 00007f43748fa94c 00007f43748f9d86 00007f43748f972f
0925 03:04:30.265142 (+ 271us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":706836096}
I20260925 03:04:30.268883 25637 consensus_queue.cc:1046] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260925 03:04:30.270325 25637 consensus_queue.cc:1035] T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260925 03:04:30.237707 25793 raft_consensus.cc:3037] T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:30.215628 24547 rpcz_store.cc:276] 0925 03:04:28.916403 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:28.997519 (+ 81116us) service_pool.cc:227] Handling call
0925 03:04:29.001158 (+ 3639us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:04:29.038259 (+ 37101us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:29.038294 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:04:29.038758 (+ 464us) log.cc:842] Serialized 26 byte log entry
0925 03:04:29.795326 (+756568us) raft_consensus.cc:1631] Marking committed up to 2
0925 03:04:30.044824 (+249498us) log.cc:842] Serialized 14 byte log entry
0925 03:04:30.141801 (+ 96977us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:30.182287 (+ 40486us) spinlock_profiling.cc:243] Waited 26.2 ms on lock 0x7b54000faad4. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d0f51 00007f43764cf99e 00007f437b4112e9 00007f4374af7684 00007f4374af7612 00007f4374af7541 00007f4374af74bc 00007f4374af6772 00007f43748fa94c 00007f43748f9d86
0925 03:04:30.182340 (+ 53us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:04:30.182432 (+ 92us) raft_consensus.cc:1671] finished
0925 03:04:30.182564 (+ 132us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:30.182723 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":59774,"prepare.queue_time_us":367959,"prepare.run_cpu_time_us":60,"prepare.run_wall_time_us":60,"raft.queue_time_us":129288,"raft.run_cpu_time_us":313,"raft.run_wall_time_us":8974,"spinlock_wait_cycles":60282240,"thread_start_us":958387,"threads_started":2,"wal-append.queue_time_us":818922}
I20260925 03:04:30.272996 24622 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RunLeaderElection from 127.0.0.1:38570 (request call id 1216) took 1159 ms. Trace:
I20260925 03:04:30.274132 24614 raft_consensus.cc:2454] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 3.
I20260925 03:04:30.270725 25978 raft_consensus.cc:3032] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c [term 3 LEADER]: Stepping down as leader of term 3
I20260925 03:04:30.257886 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9c33fd100ff44c1a3247319a324a044" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:30.275331 24622 rpcz_store.cc:276] 0925 03:04:29.102146 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:29.102311 (+ 165us) service_pool.cc:227] Handling call
0925 03:04:30.190980 (+1088669us) spinlock_profiling.cc:243] Waited 576 ms on lock 0x7b5400364554. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764c7da7 00007f437b413ee0 00007f4374b05784 00007f4374b05712 00007f4374b05641 00007f4374b055bc 00007f4374b04872 00007f43748fa94c 00007f43748f9d86 00007f43748f972f
0925 03:04:30.261404 (+ 70424us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1323791232}
I20260925 03:04:30.304214 25805 raft_consensus.cc:3032] T c88cbd39e01f44628a094af516be2833 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 LEADER]: Stepping down as leader of term 3
I20260925 03:04:30.306353 25805 raft_consensus.cc:729] T c88cbd39e01f44628a094af516be2833 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:04:30.307420 25805 consensus_queue.cc:260] T c88cbd39e01f44628a094af516be2833 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:30.308904 25950 consensus_queue.cc:1035] T 0b5df303cd6a4d59aca5c18bbbff24ed P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.030s
I20260925 03:04:30.311254 25984 raft_consensus.cc:504] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:30.314126 25984 leader_election.cc:288] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:30.297118 25942 raft_consensus.cc:504] T 5278a60f97da4e10aecd50a8d9a9a43c P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:04:30.325906 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 48562.9us
W20260925 03:04:30.331780 25900 consensus_meta.cc:287] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5: Time spent flushing consensus metadata: real 0.537s user 0.013s sys 0.000s
I20260925 03:04:30.293275 25978 raft_consensus.cc:729] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:30.357262 24543 raft_consensus.cc:3037] T 367277daf8d04778b3fa4ed94d71580d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:30.338994 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be4e0aa984f4f2b81a58e382e434ad5" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:30.308276 25931 raft_consensus.cc:504] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:30.321400 24473 raft_consensus.cc:2454] T 1414dcfe53f64a78ae413531eb8ec117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 2.
I20260925 03:04:30.332412 25805 raft_consensus.cc:3037] T c88cbd39e01f44628a094af516be2833 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:30.338249 24548 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 597) took 1197 ms. Trace:
I20260925 03:04:30.339118 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceab6b0fccd54e909a64f66fe81a9a1f" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:30.339520 25890 raft_consensus.cc:2726] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260925 03:04:30.341099 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d89396576884f7c8901dba54f783947" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:30.342051 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "420163a6cdce412ebc21c25fe3835638" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
W20260925 03:04:30.348986 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 70735.6us
I20260925 03:04:30.350355 25950 raft_consensus.cc:482] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:30.352219 24585 leader_election.cc:302] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:04:30.404539 24546 raft_consensus.cc:3037] T 0be4e0aa984f4f2b81a58e382e434ad5 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Advancing to term 3
I20260925 03:04:30.356249 25942 leader_election.cc:288] T 5278a60f97da4e10aecd50a8d9a9a43c P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
W20260925 03:04:30.408604 24436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.4:43979, user_credentials={real_user=slave}} blocked reactor thread for 60210.3us
I20260925 03:04:30.412676 25953 consensus_queue.cc:1046] T 0be4e0aa984f4f2b81a58e382e434ad5 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260925 03:04:30.420158 24543 raft_consensus.cc:2454] T 367277daf8d04778b3fa4ed94d71580d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 4.
I20260925 03:04:30.425263 24615 raft_consensus.cc:1226] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260925 03:04:30.427194 24542 raft_consensus.cc:3037] T 3d89396576884f7c8901dba54f783947 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Advancing to term 3
I20260925 03:04:30.413623 25832 consensus_queue.cc:1046] T 3d89396576884f7c8901dba54f783947 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260925 03:04:30.381292 24548 rpcz_store.cc:276] 0925 03:04:29.140990 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:29.141109 (+ 119us) service_pool.cc:227] Handling call
0925 03:04:29.141654 (+ 545us) raft_consensus.cc:1457] Updating replica for 2 ops
0925 03:04:29.175063 (+ 33409us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:29.175093 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0925 03:04:29.182129 (+ 7036us) log.cc:842] Serialized 582 byte log entry
0925 03:04:29.800251 (+618122us) raft_consensus.cc:1631] Marking committed up to 4
0925 03:04:29.889732 (+ 89481us) log.cc:842] Serialized 14 byte log entry
0925 03:04:30.161512 (+271780us) log.cc:842] Serialized 14 byte log entry
0925 03:04:30.337692 (+176180us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:30.337823 (+ 131us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:04:30.337886 (+ 63us) raft_consensus.cc:1671] finished
0925 03:04:30.337898 (+ 12us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:30.338074 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":87588,"prepare.queue_time_us":258276,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":46,"raft.queue_time_us":103702,"raft.run_cpu_time_us":690,"raft.run_wall_time_us":61370,"thread_start_us":708857,"threads_started":2,"wal-append.queue_time_us":728039}
I20260925 03:04:30.432020 24514 leader_election.cc:302] T b4ebd2b043024a548005c8b6b00b0449 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:04:30.437711 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5278a60f97da4e10aecd50a8d9a9a43c" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:30.414297 25953 consensus_queue.cc:1046] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.020s
I20260925 03:04:30.429141 25900 raft_consensus.cc:3032] T 0be4e0aa984f4f2b81a58e382e434ad5 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 LEADER]: Stepping down as leader of term 1
I20260925 03:04:30.439931 25931 leader_election.cc:288] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:30.440177 25979 raft_consensus.cc:2726] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260925 03:04:30.455101 24585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.4:43979, user_credentials={real_user=slave}} blocked reactor thread for 103244us
I20260925 03:04:30.456158 24511 leader_election.cc:398] T 0b5df303cd6a4d59aca5c18bbbff24ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 pre-election: Vote denied by peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105) with higher term. Message: Invalid argument: T 0b5df303cd6a4d59aca5c18bbbff24ed P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 3. Current term is 4.
I20260925 03:04:30.469169 24511 leader_election.cc:401] T 0b5df303cd6a4d59aca5c18bbbff24ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20260925 03:04:30.455956 25978 consensus_queue.cc:260] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:30.473358 25978 raft_consensus.cc:3037] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:30.476925 24541 raft_consensus.cc:1261] T 0b5df303cd6a4d59aca5c18bbbff24ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260925 03:04:30.477571 25783 raft_consensus.cc:3032] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 LEADER]: Stepping down as leader of term 3
I20260925 03:04:30.478006 25783 raft_consensus.cc:729] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:04:30.479108 25783 consensus_queue.cc:260] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:30.479229 25987 consensus_queue.cc:260] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:30.437000 25950 raft_consensus.cc:504] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:30.485553 24436 leader_election.cc:302] T 37a157996e944db2906110d87ecb0c54 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:30.487089 24436 leader_election.cc:302] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:30.487540 25960 consensus_queue.cc:1046] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260925 03:04:30.449246 24586 leader_election.cc:302] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:30.488803 25960 consensus_queue.cc:1046] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260925 03:04:30.489367 25931 raft_consensus.cc:2726] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260925 03:04:30.489921 25832 raft_consensus.cc:2781] T 37a157996e944db2906110d87ecb0c54 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260925 03:04:30.490384 25832 raft_consensus.cc:482] T 37a157996e944db2906110d87ecb0c54 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting leader election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:30.490777 25832 raft_consensus.cc:3037] T 37a157996e944db2906110d87ecb0c54 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:30.490861 25957 raft_consensus.cc:2781] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260925 03:04:30.491269 25957 raft_consensus.cc:482] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting leader election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:30.491588 25957 raft_consensus.cc:3037] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:30.480517 25783 raft_consensus.cc:3037] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:30.492993 25897 raft_consensus.cc:3032] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 LEADER]: Stepping down as leader of term 3
I20260925 03:04:30.458801 25885 raft_consensus.cc:482] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:30.465457 25819 raft_consensus.cc:482] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:30.500294 24542 raft_consensus.cc:2454] T 3d89396576884f7c8901dba54f783947 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 3.
I20260925 03:04:30.457664 25938 raft_consensus.cc:3032] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 LEADER]: Stepping down as leader of term 3
W20260925 03:04:30.489521 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 49954.7us
I20260925 03:04:30.495756 25897 raft_consensus.cc:729] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:04:30.497408 24467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be4e0aa984f4f2b81a58e382e434ad5" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:30.449095 25900 raft_consensus.cc:729] T 0be4e0aa984f4f2b81a58e382e434ad5 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:04:30.480437 25987 raft_consensus.cc:3037] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:30.505048 25851 raft_consensus.cc:2726] T 0b5df303cd6a4d59aca5c18bbbff24ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105) with higher term. Message: Invalid argument: T 0b5df303cd6a4d59aca5c18bbbff24ed P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 3. Current term is 4.
I20260925 03:04:30.505748 25819 raft_consensus.cc:504] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
W20260925 03:04:30.504904 24511 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 63317.9us
I20260925 03:04:30.508558 24586 leader_election.cc:302] T 3d89396576884f7c8901dba54f783947 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:30.510979 24546 raft_consensus.cc:2454] T 0be4e0aa984f4f2b81a58e382e434ad5 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 3.
I20260925 03:04:30.473345 25865 raft_consensus.cc:3032] T 3d89396576884f7c8901dba54f783947 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 LEADER]: Stepping down as leader of term 1
I20260925 03:04:30.511512 24464 raft_consensus.cc:2454] T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 2.
I20260925 03:04:30.485399 25804 raft_consensus.cc:504] T 43366a0a2f9d4344bf4e37ffd290ef8a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:30.507326 25938 raft_consensus.cc:729] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:04:30.521867 25741 consensus_queue.cc:1046] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20260925 03:04:30.522291 25925 raft_consensus.cc:2726] T 3d89396576884f7c8901dba54f783947 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260925 03:04:30.513643 25885 raft_consensus.cc:504] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:04:30.518931 25163 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 128524us
I20260925 03:04:30.523521 25900 consensus_queue.cc:260] T 0be4e0aa984f4f2b81a58e382e434ad5 P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:30.532812 25947 raft_consensus.cc:3032] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 LEADER]: Stepping down as leader of term 3
I20260925 03:04:30.533264 25900 raft_consensus.cc:3037] T 0be4e0aa984f4f2b81a58e382e434ad5 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:30.512012 25890 raft_consensus.cc:2726] T b4ebd2b043024a548005c8b6b00b0449 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260925 03:04:30.533267 25947 raft_consensus.cc:729] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:04:30.538488 25947 consensus_queue.cc:260] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:30.524817 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d89396576884f7c8901dba54f783947" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:30.513505 25931 raft_consensus.cc:482] T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:30.516830 25865 raft_consensus.cc:729] T 3d89396576884f7c8901dba54f783947 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:04:30.527784 24545 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 610) took 1244 ms. Trace:
I20260925 03:04:30.542111 25938 consensus_queue.cc:260] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:30.545336 25938 raft_consensus.cc:3037] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:30.533630 25950 leader_election.cc:288] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:30.537192 25890 consensus_queue.cc:1046] T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
W20260925 03:04:30.542116 24437 leader_election.cc:334] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389): Timed out: RequestConsensusVote RPC to 127.22.193.2:36389 timed out after 2.052s (SENT)
I20260925 03:04:30.579536 24437 leader_election.cc:302] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:30.579906 25851 raft_consensus.cc:3032] T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 LEADER]: Stepping down as leader of term 3
I20260925 03:04:30.532639 25925 raft_consensus.cc:482] T a4abe0ea757e4886be4945c5001e529a P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:30.580307 25851 raft_consensus.cc:729] T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:04:30.580732 25925 raft_consensus.cc:504] T a4abe0ea757e4886be4945c5001e529a P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:30.581338 25851 consensus_queue.cc:260] T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:30.582588 25832 raft_consensus.cc:504] T 37a157996e944db2906110d87ecb0c54 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:30.583087 25197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:30.582680 25851 raft_consensus.cc:3037] T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:30.542441 25931 raft_consensus.cc:504] T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:30.537178 25897 consensus_queue.cc:260] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:30.546346 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:30.546861 25819 leader_election.cc:288] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:30.529877 25741 raft_consensus.cc:3032] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 LEADER]: Stepping down as leader of term 3
I20260925 03:04:30.531937 25195 raft_consensus.cc:1226] T 7e49248643df4131a1c42e74244e05c2 P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 3. Current term is 4. Ops: []
W20260925 03:04:30.548885 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 122174us
I20260925 03:04:30.548096 25191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ac466793d094325ace224708b2158ed" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
W20260925 03:04:30.592921 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 66255.7us
I20260925 03:04:30.593966 25198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71f461a6e2ff434993b78a2d077f0abd" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:30.594229 25741 raft_consensus.cc:729] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:04:30.595398 25741 consensus_queue.cc:260] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:30.545110 25865 consensus_queue.cc:260] T 3d89396576884f7c8901dba54f783947 P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:30.544525 24545 rpcz_store.cc:276] 0925 03:04:29.282396 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:29.884228 (+601832us) service_pool.cc:227] Handling call
0925 03:04:29.894316 (+ 10088us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:04:30.093177 (+198861us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:30.093222 (+ 45us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:04:30.093921 (+ 699us) log.cc:842] Serialized 26 byte log entry
0925 03:04:30.282646 (+188725us) raft_consensus.cc:1631] Marking committed up to 2
0925 03:04:30.303162 (+ 20516us) log.cc:842] Serialized 14 byte log entry
0925 03:04:30.316178 (+ 13016us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:30.322085 (+ 5907us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:04:30.526185 (+204100us) raft_consensus.cc:1671] finished
0925 03:04:30.526203 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:30.526336 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":210968,"prepare.queue_time_us":20348,"prepare.run_cpu_time_us":62,"prepare.run_wall_time_us":63,"raft.queue_time_us":278023,"raft.run_cpu_time_us":285,"raft.run_wall_time_us":37916,"thread_start_us":188266,"threads_started":1,"wal-append.queue_time_us":338025}
I20260925 03:04:30.553507 25765 raft_consensus.cc:482] T 550118bc1a0641ce9803bdbf43f2069f P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:30.555117 24586 leader_election.cc:302] T cc2df40db08649c9a75e90000b878559 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:30.532599 25885 leader_election.cc:288] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:30.557722 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceab6b0fccd54e909a64f66fe81a9a1f" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:30.557945 25982 raft_consensus.cc:482] T e92f377a79354a6ebb4eec67303c13e3 P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:30.558952 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5278a60f97da4e10aecd50a8d9a9a43c" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:30.559510 25196 raft_consensus.cc:1261] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260925 03:04:30.561564 24465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5f9b503b0ab43999d582015a367441e" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:30.558980 24464 raft_consensus.cc:1226] T 293ef69caf514ac98510a68495f54eda P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Rejecting Update request from peer 1bc1f2e62ab44af995165bd5116a9d43 for earlier term 2. Current term is 3. Ops: []
I20260925 03:04:30.541147 25947 raft_consensus.cc:3037] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:30.564370 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35992316cb8d4c028333388eff02ecde" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:30.606364 24437 leader_election.cc:302] T 7b8e0d260a884ecdb20f79de46c36b19 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:04:30.564599 25804 leader_election.cc:288] T 43366a0a2f9d4344bf4e37ffd290ef8a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:30.566874 24588 leader_election.cc:302] T 1414dcfe53f64a78ae413531eb8ec117 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:30.572652 25190 raft_consensus.cc:1226] T 8ac466793d094325ace224708b2158ed P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 2. Current term is 3. Ops: []
I20260925 03:04:30.573277 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abd5b8dc1b7e49e3aa9af900d73ff0e7" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:30.573431 25957 raft_consensus.cc:504] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:30.580302 25793 raft_consensus.cc:504] T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:30.586903 24371 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39382 (request call id 189) took 6112 ms. Trace:
I20260925 03:04:30.587199 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37a157996e944db2906110d87ecb0c54" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:30.590544 25197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15861cd28e354598a3c6bfc4337c34bb" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:30.590857 25940 raft_consensus.cc:2726] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260925 03:04:30.591888 25192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4abe0ea757e4886be4945c5001e529a" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:30.587833 25925 leader_election.cc:288] T a4abe0ea757e4886be4945c5001e529a P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:30.598461 25865 raft_consensus.cc:3037] T 3d89396576884f7c8901dba54f783947 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:30.588920 25832 leader_election.cc:288] T 37a157996e944db2906110d87ecb0c54 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:30.633482 24468 raft_consensus.cc:2454] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 3.
I20260925 03:04:30.616822 25765 raft_consensus.cc:504] T 550118bc1a0641ce9803bdbf43f2069f P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:30.634603 25793 leader_election.cc:288] T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:30.635329 25765 leader_election.cc:288] T 550118bc1a0641ce9803bdbf43f2069f P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:30.635721 24464 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a7aa47c48404465b94a36fd819fe3c7a"
dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
mode: GRACEFUL
new_leader_uuid: "eeab93abfdb141a895002ec9f312296c"
from {username='slave'} at 127.0.0.1:38832
I20260925 03:04:30.636718 25192 raft_consensus.cc:2454] T a4abe0ea757e4886be4945c5001e529a P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 1.
I20260925 03:04:30.614306 25960 raft_consensus.cc:2781] T 7b8e0d260a884ecdb20f79de46c36b19 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260925 03:04:30.637591 25960 raft_consensus.cc:482] T 7b8e0d260a884ecdb20f79de46c36b19 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:30.632431 25197 raft_consensus.cc:2379] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 4: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:04:30.637939 25960 raft_consensus.cc:3037] T 7b8e0d260a884ecdb20f79de46c36b19 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Advancing to term 3
W20260925 03:04:30.636041 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 69262us
I20260925 03:04:30.640216 24588 leader_election.cc:302] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
W20260925 03:04:30.641275 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 86294.9us
I20260925 03:04:30.641295 24585 leader_election.cc:302] T a4abe0ea757e4886be4945c5001e529a P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:30.642433 24586 leader_election.cc:302] T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:30.636636 25885 raft_consensus.cc:2781] T 1414dcfe53f64a78ae413531eb8ec117 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Leader election won for term 2
I20260925 03:04:30.615861 24466 raft_consensus.cc:2454] T ceab6b0fccd54e909a64f66fe81a9a1f P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 3.
I20260925 03:04:30.643316 25885 raft_consensus.cc:686] T 1414dcfe53f64a78ae413531eb8ec117 P eeab93abfdb141a895002ec9f312296c [term 2 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:30.643587 25897 raft_consensus.cc:3037] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:30.643909 25925 raft_consensus.cc:2726] T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260925 03:04:30.643121 25950 raft_consensus.cc:2781] T a4abe0ea757e4886be4945c5001e529a P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 03:04:30.644680 25950 raft_consensus.cc:482] T a4abe0ea757e4886be4945c5001e529a P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting leader election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:30.645125 25950 raft_consensus.cc:3037] T a4abe0ea757e4886be4945c5001e529a P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:30.644459 24586 leader_election.cc:302] T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:30.646589 25925 raft_consensus.cc:3037] T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:30.641248 25952 raft_consensus.cc:2781] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260925 03:04:30.649187 25952 raft_consensus.cc:482] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting leader election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:30.649256 24514 leader_election.cc:302] T edcf6e6a422d4d198d4fbb0f7d3d845d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:30.633033 24469 raft_consensus.cc:2379] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c in current term 2: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:04:30.645592 25885 consensus_queue.cc:237] T 1414dcfe53f64a78ae413531eb8ec117 P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:30.616982 24371 rpcz_store.cc:276] 0925 03:04:24.474819 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:24.474975 (+ 156us) service_pool.cc:227] Handling call
0925 03:04:27.066483 (+2591508us) spinlock_profiling.cc:243] Waited 1.65 s on lock 0x7b54001509d4. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455af0 00007f43764d8f2a 00007f437c42ad6c 00007f437c524c3e 00007f437a836fb4 00007f437a836f42 00007f437a836e71 00007f437a836dec 00007f437a8360a2 00007f43748fa94c 00007f43748f9d86
0925 03:04:30.586688 (+3520205us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0925 03:04:29.305303 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0925 03:04:29.411118 (+105815us) write_op.cc:432] Acquiring schema lock in shared mode
0925 03:04:29.411159 (+ 41us) write_op.cc:435] Acquired schema lock
0925 03:04:29.411174 (+ 15us) tablet.cc:660] Decoding operations
0925 03:04:29.467836 (+ 56662us) write_op.cc:620] Acquiring the partition lock for write op
0925 03:04:29.470093 (+ 2257us) write_op.cc:641] Partition lock acquired for write op
0925 03:04:29.474629 (+ 4536us) tablet.cc:683] Acquiring locks for 5 operations
0925 03:04:29.587659 (+113030us) tablet.cc:699] Row locks acquired
0925 03:04:29.596439 (+ 8780us) write_op.cc:260] PREPARE: finished
0925 03:04:29.623061 (+ 26622us) write_op.cc:270] Start()
0925 03:04:29.638180 (+ 15119us) write_op.cc:276] Timestamp: P: 1790305469622983 usec, L: 0
0925 03:04:29.643069 (+ 4889us) op_driver.cc:331] REPLICATION: starting
0925 03:04:29.702191 (+ 59122us) log.cc:842] Serialized 2078 byte log entry
0925 03:04:30.246719 (+544528us) op_driver.cc:444] REPLICATION: finished
0925 03:04:30.436888 (+190169us) write_op.cc:301] APPLY: starting
0925 03:04:30.440726 (+ 3838us) tablet.cc:1365] starting BulkCheckPresence
0925 03:04:30.440981 (+ 255us) tablet.cc:1368] finished BulkCheckPresence
0925 03:04:30.441999 (+ 1018us) tablet.cc:1370] starting ApplyRowOperation cycle
0925 03:04:30.443236 (+ 1237us) tablet.cc:1381] finished ApplyRowOperation cycle
0925 03:04:30.443636 (+ 400us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=5
0925 03:04:30.443728 (+ 92us) write_op.cc:312] APPLY: finished
0925 03:04:30.444212 (+ 484us) log.cc:842] Serialized 46 byte log entry
0925 03:04:30.444468 (+ 256us) write_op.cc:489] Releasing partition, row and schema locks
0925 03:04:30.449759 (+ 5291us) write_op.cc:454] Released schema lock
0925 03:04:30.500029 (+ 50270us) write_op.cc:341] FINISH: Updating metrics
0925 03:04:30.555955 (+ 55926us) write_op.cc:489] Releasing partition, row and schema locks
0925 03:04:30.556005 (+ 50us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1441047,"reported_tablets":10,"spinlock_wait_cycles":3785954048,"child_traces":[["op",{"apply.queue_time_us":179414,"apply.run_cpu_time_us":4460,"apply.run_wall_time_us":127229,"num_ops":5,"prepare.queue_time_us":479344,"prepare.run_cpu_time_us":65491,"prepare.run_wall_time_us":673442,"raft.queue_time_us":254393,"raft.run_cpu_time_us":68711,"raft.run_wall_time_us":174914,"replication_time_us":603447,"thread_start_us":892957,"threads_started":4,"wal-append.queue_time_us":284805}]]}
I20260925 03:04:30.653281 25197 raft_consensus.cc:1226] T 04e068a0f80941dc9138cee063f53bab P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 LEADER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 1. Current term is 2. Ops: []
I20260925 03:04:30.653167 24586 leader_election.cc:302] T 420163a6cdce412ebc21c25fe3835638 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:30.654978 25979 raft_consensus.cc:2726] T 420163a6cdce412ebc21c25fe3835638 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260925 03:04:30.654786 25192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "293ef69caf514ac98510a68495f54eda" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:30.655514 24464 raft_consensus.cc:595] T a7aa47c48404465b94a36fd819fe3c7a P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 LEADER]: Received request to transfer leadership to TS eeab93abfdb141a895002ec9f312296c
I20260925 03:04:30.655846 25192 raft_consensus.cc:2454] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 3.
I20260925 03:04:30.642275 25931 leader_election.cc:288] T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
W20260925 03:04:30.657024 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 55941.3us
I20260925 03:04:30.658336 23300 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260925 03:04:30.658969 24437 leader_election.cc:302] T 25102ba08e4e49388f6176badcd10a25 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:30.659958 25940 raft_consensus.cc:2726] T 25102ba08e4e49388f6176badcd10a25 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 03:04:30.617431 24621 raft_consensus.cc:3037] T 37a157996e944db2906110d87ecb0c54 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:30.660039 24586 leader_election.cc:302] T 367277daf8d04778b3fa4ed94d71580d P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:04:30.617166 24470 raft_consensus.cc:2379] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c in current term 4: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:04:30.649648 25952 raft_consensus.cc:3037] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:30.666169 24614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "550118bc1a0641ce9803bdbf43f2069f" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:30.666858 24614 raft_consensus.cc:2454] T 550118bc1a0641ce9803bdbf43f2069f P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 2.
I20260925 03:04:30.665194 23300 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260925 03:04:30.667763 25765 consensus_queue.cc:1035] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260925 03:04:30.668892 23300 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260925 03:04:30.664994 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71f461a6e2ff434993b78a2d077f0abd" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:30.671226 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b627ecb4a5384ef392aa695e06599c39" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:30.672492 24464 raft_consensus.cc:2454] T b627ecb4a5384ef392aa695e06599c39 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 3.
I20260925 03:04:30.673631 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48ca6ccd98194cf2af7b5278d15bf4ea" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:30.674415 25931 raft_consensus.cc:2781] T 367277daf8d04778b3fa4ed94d71580d P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader election won for term 4
I20260925 03:04:30.673846 25805 raft_consensus.cc:2726] T edcf6e6a422d4d198d4fbb0f7d3d845d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 03:04:30.675024 25931 raft_consensus.cc:686] T 367277daf8d04778b3fa4ed94d71580d P eeab93abfdb141a895002ec9f312296c [term 4 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:30.674286 24367 catalog_manager.cc:5495] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: HEALTHY } } }
I20260925 03:04:30.676064 25931 consensus_queue.cc:237] T 367277daf8d04778b3fa4ed94d71580d P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } }
I20260925 03:04:30.677007 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f68804eb5a244d87bb395d50683c6012" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:30.673622 24514 leader_election.cc:302] T f22ddd7d1c444dc58f45a1fb0b3e02bc P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:30.681738 24614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b030d84c670b48d5bff337ccc0c77626" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:30.682632 24467 raft_consensus.cc:3037] T 0be4e0aa984f4f2b81a58e382e434ad5 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:30.683074 24614 raft_consensus.cc:2379] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 4: Already voted for candidate eeab93abfdb141a895002ec9f312296c in this term.
I20260925 03:04:30.683641 25805 raft_consensus.cc:2726] T f22ddd7d1c444dc58f45a1fb0b3e02bc P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260925 03:04:30.685747 24614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5f9b503b0ab43999d582015a367441e" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:30.686514 24614 raft_consensus.cc:2354] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 3. Current term is 4.
I20260925 03:04:30.687476 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be4e0aa984f4f2b81a58e382e434ad5" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:30.688378 24613 raft_consensus.cc:2354] T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 2. Current term is 3.
I20260925 03:04:30.688962 25979 consensus_queue.cc:1035] T 0b5df303cd6a4d59aca5c18bbbff24ed P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260925 03:04:30.689463 24618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9357d144aa49789faaf31fe5e6bf89" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:30.691345 24586 leader_election.cc:302] T 3d89396576884f7c8901dba54f783947 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:30.692741 25979 raft_consensus.cc:2781] T 3d89396576884f7c8901dba54f783947 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader election won for term 3
I20260925 03:04:30.692641 25793 consensus_queue.cc:1046] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260925 03:04:30.693588 25979 raft_consensus.cc:686] T 3d89396576884f7c8901dba54f783947 P eeab93abfdb141a895002ec9f312296c [term 3 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:30.693998 25805 raft_consensus.cc:3032] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 LEADER]: Stepping down as leader of term 3
I20260925 03:04:30.694388 25805 raft_consensus.cc:729] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:04:30.611428 25982 raft_consensus.cc:504] T e92f377a79354a6ebb4eec67303c13e3 P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:30.695168 24514 leader_election.cc:398] T 0b5df303cd6a4d59aca5c18bbbff24ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187) with higher term. Message: Invalid argument: T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 3. Current term is 4.
I20260925 03:04:30.694633 25979 consensus_queue.cc:237] T 3d89396576884f7c8901dba54f783947 P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:30.613229 25819 raft_consensus.cc:2726] T cc2df40db08649c9a75e90000b878559 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260925 03:04:30.701145 24618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "100c113a9fd84cc8adf2ec7b33ea0093" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:30.702356 24618 raft_consensus.cc:2379] T 100c113a9fd84cc8adf2ec7b33ea0093 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 2: Already voted for candidate eeab93abfdb141a895002ec9f312296c in this term.
I20260925 03:04:30.697584 25957 leader_election.cc:288] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:30.703218 25805 consensus_queue.cc:260] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:30.616051 25741 raft_consensus.cc:3037] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:30.708200 24436 leader_election.cc:302] T 550118bc1a0641ce9803bdbf43f2069f P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:30.709290 24588 leader_election.cc:302] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:30.709689 24467 raft_consensus.cc:2454] T 0be4e0aa984f4f2b81a58e382e434ad5 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 3.
I20260925 03:04:30.704417 25950 raft_consensus.cc:504] T a4abe0ea757e4886be4945c5001e529a P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:30.698028 24514 leader_election.cc:302] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:30.712545 24471 raft_consensus.cc:3037] T 3d89396576884f7c8901dba54f783947 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:30.714303 24588 leader_election.cc:302] T 5278a60f97da4e10aecd50a8d9a9a43c P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:30.715533 24618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60d45b7d467d4e1ab92367f6d7d52b15" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:30.716228 24618 raft_consensus.cc:3037] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:30.715204 24586 leader_election.cc:302] T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:30.717130 24588 leader_election.cc:302] T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:30.719702 25978 raft_consensus.cc:3037] T 5278a60f97da4e10aecd50a8d9a9a43c P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:30.721166 25925 raft_consensus.cc:2726] T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260925 03:04:30.713265 25819 raft_consensus.cc:2781] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260925 03:04:30.722265 25819 raft_consensus.cc:482] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:30.722589 25819 raft_consensus.cc:3037] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:30.704646 25805 raft_consensus.cc:3037] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:30.727171 24471 raft_consensus.cc:2454] T 3d89396576884f7c8901dba54f783947 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 3.
I20260925 03:04:30.728832 25890 raft_consensus.cc:2726] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260925 03:04:30.729344 25942 raft_consensus.cc:2781] T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader election won for term 3
I20260925 03:04:30.730130 25982 leader_election.cc:288] T e92f377a79354a6ebb4eec67303c13e3 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:30.676231 24367 catalog_manager.cc:5495] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 reported cstate change: config changed from index 2 to 3, 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) changed from NON_VOTER to VOTER, no longer has a pending config: VOTER 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1), VOTER a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2), VOTER eeab93abfdb141a895002ec9f312296c (127.22.193.3), VOTER 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4). New cstate: current_term: 3 leader_uuid: "869b0a8575b54d08ae2817fad5dd42d5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:30.732110 24437 leader_election.cc:302] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:04:30.733515 24511 leader_election.cc:302] T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:04:30.733829 25925 raft_consensus.cc:2781] T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260925 03:04:30.734506 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e92f377a79354a6ebb4eec67303c13e3" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:30.737349 25197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4abe0ea757e4886be4945c5001e529a" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43"
I20260925 03:04:30.737691 25978 raft_consensus.cc:2726] T 5278a60f97da4e10aecd50a8d9a9a43c P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 03:04:30.738108 24621 raft_consensus.cc:2454] T 37a157996e944db2906110d87ecb0c54 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 4.
I20260925 03:04:30.738718 25952 raft_consensus.cc:504] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:30.739215 25960 raft_consensus.cc:504] T 7b8e0d260a884ecdb20f79de46c36b19 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:30.713919 25957 raft_consensus.cc:2781] T 550118bc1a0641ce9803bdbf43f2069f P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260925 03:04:30.746222 24614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f1fac028a144058b92331c0d9b44ebf" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:30.737830 25950 leader_election.cc:288] T a4abe0ea757e4886be4945c5001e529a P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:30.751256 25197 raft_consensus.cc:3037] T a4abe0ea757e4886be4945c5001e529a P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:30.752085 24436 leader_election.cc:302] T 37a157996e944db2906110d87ecb0c54 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:30.752818 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21c43af5e6844fdaacc8c625fa01fd8d" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:30.755851 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "550118bc1a0641ce9803bdbf43f2069f" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:30.745261 25925 raft_consensus.cc:482] T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting leader election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:30.746088 25942 raft_consensus.cc:686] T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c [term 3 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:30.760427 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abd5b8dc1b7e49e3aa9af900d73ff0e7" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:30.760864 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ac466793d094325ace224708b2158ed" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:30.761554 24619 raft_consensus.cc:2379] T 550118bc1a0641ce9803bdbf43f2069f P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 2: Already voted for candidate eeab93abfdb141a895002ec9f312296c in this term.
I20260925 03:04:30.763253 24621 raft_consensus.cc:1226] T 442d4a29b1d743599f4b6b4871e26668 P eeab93abfdb141a895002ec9f312296c [term 4 LEADER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 3. Current term is 4. Ops: []
I20260925 03:04:30.752650 24613 raft_consensus.cc:2454] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 2.
I20260925 03:04:30.752637 25897 raft_consensus.cc:2781] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260925 03:04:30.765266 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5f9b503b0ab43999d582015a367441e" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:30.754245 25957 raft_consensus.cc:482] T 550118bc1a0641ce9803bdbf43f2069f P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260925 03:04:30.766296 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 68356.3us
I20260925 03:04:30.768584 24545 raft_consensus.cc:3037] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:30.758380 25938 raft_consensus.cc:2781] T 37a157996e944db2906110d87ecb0c54 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader election won for term 4
I20260925 03:04:30.762087 25925 raft_consensus.cc:3037] T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:30.771076 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b627ecb4a5384ef392aa695e06599c39" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:30.761082 25890 raft_consensus.cc:2781] T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 03:04:30.771217 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1414dcfe53f64a78ae413531eb8ec117" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:30.774138 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc2df40db08649c9a75e90000b878559" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:30.768204 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35992316cb8d4c028333388eff02ecde" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:30.769008 25957 raft_consensus.cc:3037] T 550118bc1a0641ce9803bdbf43f2069f P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:30.767410 25897 raft_consensus.cc:482] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:30.786684 25162 leader_election.cc:302] T e92f377a79354a6ebb4eec67303c13e3 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:30.787622 25982 raft_consensus.cc:2781] T e92f377a79354a6ebb4eec67303c13e3 P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260925 03:04:30.787689 25960 leader_election.cc:288] T 7b8e0d260a884ecdb20f79de46c36b19 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:30.774641 24539 raft_consensus.cc:3032] T 442d4a29b1d743599f4b6b4871e26668 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 LEADER]: Stepping down as leader of term 3
I20260925 03:04:30.786011 24514 leader_election.cc:302] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:04:30.790629 24621 raft_consensus.cc:2379] T cc2df40db08649c9a75e90000b878559 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 2: Already voted for candidate eeab93abfdb141a895002ec9f312296c in this term.
I20260925 03:04:30.784329 24367 catalog_manager.cc:5495] T 37a157996e944db2906110d87ecb0c54 P a04a37fc4aa7460cbfdbeeb4e451e1f5 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: UNKNOWN } } }
I20260925 03:04:30.776954 25938 raft_consensus.cc:686] T 37a157996e944db2906110d87ecb0c54 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 LEADER]: Becoming Leader. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
W20260925 03:04:30.792587 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 60597.6us
I20260925 03:04:30.788264 25982 raft_consensus.cc:482] T e92f377a79354a6ebb4eec67303c13e3 P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:30.782047 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4abe0ea757e4886be4945c5001e529a" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:30.783828 25897 raft_consensus.cc:3037] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:30.796972 24511 leader_election.cc:302] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:04:30.785297 25890 raft_consensus.cc:686] T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 LEADER]: Becoming Leader. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:04:30.800865 25890 consensus_queue.cc:237] T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:30.802402 24367 catalog_manager.cc:5495] T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "869b0a8575b54d08ae2817fad5dd42d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } }
I20260925 03:04:30.776152 25793 consensus_queue.cc:1046] T 8ac466793d094325ace224708b2158ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.019s
I20260925 03:04:30.790227 24539 raft_consensus.cc:729] T 442d4a29b1d743599f4b6b4871e26668 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:04:30.804819 24621 raft_consensus.cc:1226] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 3. Current term is 4. Ops: []
I20260925 03:04:30.803426 25851 raft_consensus.cc:2726] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260925 03:04:30.796391 25982 raft_consensus.cc:3037] T e92f377a79354a6ebb4eec67303c13e3 P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:30.809444 25960 raft_consensus.cc:482] T cc2df40db08649c9a75e90000b878559 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:30.810057 25987 raft_consensus.cc:482] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:30.818980 25987 raft_consensus.cc:504] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:30.820360 24539 consensus_queue.cc:260] T 442d4a29b1d743599f4b6b4871e26668 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:30.821148 25987 leader_election.cc:288] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:30.821676 24539 raft_consensus.cc:3037] T 442d4a29b1d743599f4b6b4871e26668 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:30.809502 24615 raft_consensus.cc:1226] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 3. Current term is 4. Ops: []
I20260925 03:04:30.819314 24540 raft_consensus.cc:1226] T cc2df40db08649c9a75e90000b878559 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 1. Current term is 2. Ops: []
I20260925 03:04:30.808185 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b8e0d260a884ecdb20f79de46c36b19" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:30.825660 24614 raft_consensus.cc:2454] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 4.
I20260925 03:04:30.826229 25799 raft_consensus.cc:3032] T 8ac466793d094325ace224708b2158ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 LEADER]: Stepping down as leader of term 2
I20260925 03:04:30.835608 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e54ca1757ee4e42956a9cebbeca4f62" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:30.836402 24472 raft_consensus.cc:2454] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 4.
I20260925 03:04:30.836678 24547 raft_consensus.cc:1226] T 550118bc1a0641ce9803bdbf43f2069f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 1. Current term is 2. Ops: []
I20260925 03:04:30.826282 25854 raft_consensus.cc:2781] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260925 03:04:30.837993 25854 raft_consensus.cc:482] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1bc1f2e62ab44af995165bd5116a9d43)
I20260925 03:04:30.838351 25854 raft_consensus.cc:3037] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:30.835852 25799 raft_consensus.cc:729] T 8ac466793d094325ace224708b2158ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:04:30.828423 25960 raft_consensus.cc:504] T cc2df40db08649c9a75e90000b878559 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:30.841275 25799 consensus_queue.cc:260] T 8ac466793d094325ace224708b2158ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:30.842782 25799 raft_consensus.cc:3037] T 8ac466793d094325ace224708b2158ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:30.843536 25938 consensus_queue.cc:237] T 37a157996e944db2906110d87ecb0c54 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:30.845623 25165 leader_election.cc:302] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5; no voters:
I20260925 03:04:30.846690 25987 raft_consensus.cc:2781] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260925 03:04:30.847092 25987 raft_consensus.cc:482] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:30.832887 25952 leader_election.cc:288] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:30.847879 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e54ca1757ee4e42956a9cebbeca4f62" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:30.848764 24613 raft_consensus.cc:2454] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 4.
I20260925 03:04:30.814489 25942 consensus_queue.cc:237] T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:30.853925 25897 raft_consensus.cc:504] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:30.856490 25960 leader_election.cc:288] T cc2df40db08649c9a75e90000b878559 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:30.858152 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc2df40db08649c9a75e90000b878559" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:30.873346 24613 raft_consensus.cc:2454] T cc2df40db08649c9a75e90000b878559 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 2.
I20260925 03:04:30.874580 24436 leader_election.cc:302] T cc2df40db08649c9a75e90000b878559 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:30.874728 25897 leader_election.cc:288] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:30.858642 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4abe0ea757e4886be4945c5001e529a" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:30.876160 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bb46bce3564841b7674e6a693bd2e6" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:30.830171 25900 raft_consensus.cc:482] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:30.862069 24540 raft_consensus.cc:1226] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 2. Current term is 3. Ops: []
I20260925 03:04:30.878196 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35992316cb8d4c028333388eff02ecde" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:30.879366 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e8af2641e124f9589c5ccfdac785f63" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:30.862565 24546 raft_consensus.cc:1226] T 48ca6ccd98194cf2af7b5278d15bf4ea P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 1. Current term is 2. Ops: []
I20260925 03:04:30.881306 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5278a60f97da4e10aecd50a8d9a9a43c" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:30.882290 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5278a60f97da4e10aecd50a8d9a9a43c" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:30.883285 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9c33fd100ff44c1a3247319a324a044" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:30.863725 24542 raft_consensus.cc:1226] T 367277daf8d04778b3fa4ed94d71580d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 3. Current term is 4. Ops: []
I20260925 03:04:30.863817 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "293ef69caf514ac98510a68495f54eda" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:30.865509 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f1fac028a144058b92331c0d9b44ebf" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:30.885008 24541 raft_consensus.cc:2379] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 in current term 4: Already voted for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in this term.
I20260925 03:04:30.885169 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74443a0a1dc34d88a2f42e3b722db405" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:30.885784 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37a157996e944db2906110d87ecb0c54" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:30.886915 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e5819c15eac4544a57e5ee6c9d4f117" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:30.887940 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f22ddd7d1c444dc58f45a1fb0b3e02bc" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:30.866176 24547 raft_consensus.cc:1226] T 21c43af5e6844fdaacc8c625fa01fd8d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 1. Current term is 2. Ops: []
W20260925 03:04:30.856359 26046 log.cc:924] Time spent T 1414dcfe53f64a78ae413531eb8ec117 P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.073s user 0.000s sys 0.001s
I20260925 03:04:30.884285 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7993f9c3e2a44f408bbf5d1e395e2306" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:30.867725 24545 raft_consensus.cc:2454] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 4.
I20260925 03:04:30.868995 25987 raft_consensus.cc:3037] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:30.869153 25854 raft_consensus.cc:504] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:30.869670 25819 raft_consensus.cc:504] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:04:30.869004 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 95680.1us
I20260925 03:04:30.892933 24543 raft_consensus.cc:1226] T 43366a0a2f9d4344bf4e37ffd290ef8a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 1. Current term is 2. Ops: []
I20260925 03:04:30.891013 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60d45b7d467d4e1ab92367f6d7d52b15" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:30.891181 25900 raft_consensus.cc:504] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:30.896369 24542 raft_consensus.cc:2454] T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 4.
I20260925 03:04:30.901798 24544 raft_consensus.cc:1226] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 3. Current term is 4. Ops: []
I20260925 03:04:30.889442 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15861cd28e354598a3c6bfc4337c34bb" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:30.890552 24546 raft_consensus.cc:2454] T 7993f9c3e2a44f408bbf5d1e395e2306 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 2.
I20260925 03:04:30.901167 25197 raft_consensus.cc:2454] T a4abe0ea757e4886be4945c5001e529a P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 2.
W20260925 03:04:30.911695 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 100561us
I20260925 03:04:30.888662 24541 raft_consensus.cc:2379] T f22ddd7d1c444dc58f45a1fb0b3e02bc P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 in current term 4: Already voted for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in this term.
I20260925 03:04:30.911586 24618 raft_consensus.cc:2454] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 4.
I20260925 03:04:30.915498 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:30.921517 24467 raft_consensus.cc:1261] T 442d4a29b1d743599f4b6b4871e26668 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260925 03:04:30.925694 24542 raft_consensus.cc:1226] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 3. Current term is 4. Ops: []
I20260925 03:04:30.920827 24436 leader_election.cc:302] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:30.905565 25957 raft_consensus.cc:504] T 550118bc1a0641ce9803bdbf43f2069f P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:30.918057 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abd5b8dc1b7e49e3aa9af900d73ff0e7" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:30.913942 25897 raft_consensus.cc:2781] T cc2df40db08649c9a75e90000b878559 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260925 03:04:30.929466 24471 raft_consensus.cc:3037] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:30.930536 24472 raft_consensus.cc:1226] T 7b8e0d260a884ecdb20f79de46c36b19 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 2. Current term is 3. Ops: []
I20260925 03:04:30.927749 24514 leader_election.cc:302] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:04:30.884445 25925 raft_consensus.cc:504] T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:30.932853 24547 raft_consensus.cc:2396] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 for term 4 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260925 03:04:30.930728 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7691c6b475428c82d6181d693d26f3" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:30.934229 24473 raft_consensus.cc:1261] T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260925 03:04:30.936177 24464 raft_consensus.cc:1261] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
I20260925 03:04:30.937626 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74443a0a1dc34d88a2f42e3b722db405" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:30.939663 24437 leader_election.cc:302] T b627ecb4a5384ef392aa695e06599c39 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:30.936056 24613 raft_consensus.cc:1226] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 3. Current term is 4. Ops: []
I20260925 03:04:30.940943 24543 raft_consensus.cc:2362] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 4. Re-sending same reply.
I20260925 03:04:30.937211 24464 pending_rounds.cc:76] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5: Aborting all ops after (but not including) 1
I20260925 03:04:30.926779 25931 raft_consensus.cc:482] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:30.944541 24514 leader_election.cc:302] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:04:30.948571 24585 leader_election.cc:302] T a4abe0ea757e4886be4945c5001e529a P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:30.930312 25897 raft_consensus.cc:482] T cc2df40db08649c9a75e90000b878559 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:30.954540 25897 raft_consensus.cc:3037] T cc2df40db08649c9a75e90000b878559 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:30.952337 25196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43"
I20260925 03:04:30.954084 25819 leader_election.cc:288] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:30.951612 25885 consensus_queue.cc:1035] T 48ca6ccd98194cf2af7b5278d15bf4ea P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260925 03:04:30.944537 24464 pending_rounds.cc:97] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20260925 03:04:30.933442 25979 consensus_queue.cc:1035] T 442d4a29b1d743599f4b6b4871e26668 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.020s
I20260925 03:04:30.932507 25938 raft_consensus.cc:2781] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader election won for term 4
I20260925 03:04:30.958855 24619 raft_consensus.cc:1226] T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 3. Current term is 4. Ops: []
I20260925 03:04:30.959633 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c23337a7cb74184a0e57b7095cb6a11" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:30.964003 24621 raft_consensus.cc:2379] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 4: Already voted for candidate eeab93abfdb141a895002ec9f312296c in this term.
I20260925 03:04:30.963339 24464 raft_consensus.cc:2866] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260925 03:04:30.960577 25984 consensus_queue.cc:1046] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260925 03:04:30.961383 25890 raft_consensus.cc:2726] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260925 03:04:30.944216 25799 raft_consensus.cc:2741] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260925 03:04:30.966426 25973 raft_consensus.cc:2781] T a4abe0ea757e4886be4945c5001e529a P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Leader election won for term 2
I20260925 03:04:30.966805 24615 raft_consensus.cc:1226] T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 3. Current term is 4. Ops: []
I20260925 03:04:30.965569 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceab6b0fccd54e909a64f66fe81a9a1f" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:30.970986 24470 raft_consensus.cc:3037] T ceab6b0fccd54e909a64f66fe81a9a1f P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:30.970161 24539 raft_consensus.cc:1261] T 442d4a29b1d743599f4b6b4871e26668 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260925 03:04:30.973784 24436 leader_election.cc:302] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: 1bc1f2e62ab44af995165bd5116a9d43, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:30.985801 25960 raft_consensus.cc:2726] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260925 03:04:30.985249 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37a157996e944db2906110d87ecb0c54" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:30.986744 24465 raft_consensus.cc:2396] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260925 03:04:30.982729 24437 leader_election.cc:302] T 442d4a29b1d743599f4b6b4871e26668 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:30.972414 25854 leader_election.cc:288] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:30.980948 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43366a0a2f9d4344bf4e37ffd290ef8a" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:30.989300 25973 raft_consensus.cc:686] T a4abe0ea757e4886be4945c5001e529a P eeab93abfdb141a895002ec9f312296c [term 2 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:30.990275 25741 raft_consensus.cc:2726] T 442d4a29b1d743599f4b6b4871e26668 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260925 03:04:30.991147 24614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e725b82985746b39c0cc8f671c95353" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:30.990302 25973 consensus_queue.cc:237] T a4abe0ea757e4886be4945c5001e529a P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:30.991983 24543 raft_consensus.cc:1261] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260925 03:04:30.990311 24437 leader_election.cc:302] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:04:30.988941 24514 leader_election.cc:302] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:04:30.967958 25931 raft_consensus.cc:504] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:30.995935 24514 leader_election.cc:302] T 48ca6ccd98194cf2af7b5278d15bf4ea P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:04:30.997313 25931 leader_election.cc:288] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:30.984143 24540 raft_consensus.cc:1261] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 5. (term mismatch)
I20260925 03:04:30.974993 24586 leader_election.cc:302] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 1bc1f2e62ab44af995165bd5116a9d43, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:31.002483 24471 raft_consensus.cc:2454] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 4.
I20260925 03:04:30.963713 25947 raft_consensus.cc:2726] T b627ecb4a5384ef392aa695e06599c39 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260925 03:04:30.980165 25900 leader_election.cc:288] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:31.008216 24540 pending_rounds.cc:76] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5: Aborting all ops after (but not including) 4
I20260925 03:04:31.007836 25799 raft_consensus.cc:2726] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260925 03:04:31.013043 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abd5b8dc1b7e49e3aa9af900d73ff0e7" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:31.014467 24472 raft_consensus.cc:1261] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260925 03:04:31.004916 25854 raft_consensus.cc:2741] T 48ca6ccd98194cf2af7b5278d15bf4ea P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20260925 03:04:30.994069 25925 leader_election.cc:288] T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
W20260925 03:04:31.020104 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 45194us
I20260925 03:04:31.020915 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b627ecb4a5384ef392aa695e06599c39" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:31.025946 24464 raft_consensus.cc:3037] T b627ecb4a5384ef392aa695e06599c39 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Advancing to term 4
W20260925 03:04:31.022068 26054 log.cc:924] Time spent T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43: Append to log took a long time: real 0.056s user 0.000s sys 0.000s
I20260925 03:04:31.026810 24544 raft_consensus.cc:1261] T 48ca6ccd98194cf2af7b5278d15bf4ea P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260925 03:04:31.022671 24618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "550118bc1a0641ce9803bdbf43f2069f" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:31.027448 24618 raft_consensus.cc:3037] T 550118bc1a0641ce9803bdbf43f2069f P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:31.008090 25938 raft_consensus.cc:686] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 LEADER]: Becoming Leader. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:04:31.024132 24511 leader_election.cc:302] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 1bc1f2e62ab44af995165bd5116a9d43, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:31.007674 25741 raft_consensus.cc:2741] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader election decision vote started in defunct term 3: won
I20260925 03:04:31.024904 24467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5aec38e66a947c681e6c2a72b05e6ca" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:31.022538 25984 raft_consensus.cc:2726] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260925 03:04:31.016868 24540 pending_rounds.cc:97] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5: Aborting uncommitted NO_OP operation due to leader change: 2.5
I20260925 03:04:31.020867 25957 leader_election.cc:288] T 550118bc1a0641ce9803bdbf43f2069f P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:31.034329 24467 raft_consensus.cc:2454] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 4.
I20260925 03:04:31.037444 24588 leader_election.cc:302] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:31.037966 24586 leader_election.cc:302] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:31.027319 25979 consensus_queue.cc:1035] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.020s
I20260925 03:04:31.048276 25885 raft_consensus.cc:3032] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [term 2 LEADER]: Stepping down as leader of term 2
I20260925 03:04:31.023171 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceab6b0fccd54e909a64f66fe81a9a1f" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:31.025350 25197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5aec38e66a947c681e6c2a72b05e6ca" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:31.047290 24586 leader_election.cc:302] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:04:31.033806 24540 raft_consensus.cc:2866] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260925 03:04:31.066572 24588 leader_election.cc:302] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:31.048681 25885 raft_consensus.cc:729] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:31.081677 24618 raft_consensus.cc:2454] T 550118bc1a0641ce9803bdbf43f2069f P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 3.
I20260925 03:04:31.055899 25984 raft_consensus.cc:3037] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:31.052814 25854 raft_consensus.cc:2726] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260925 03:04:31.061774 25925 raft_consensus.cc:2781] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader election won for term 4
I20260925 03:04:31.060667 25197 raft_consensus.cc:2454] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 4.
I20260925 03:04:31.088165 25788 raft_consensus.cc:482] T b1e85248d3b04b979dd080caeeb02fdb P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:31.089031 25938 consensus_queue.cc:237] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:31.089588 24473 raft_consensus.cc:1226] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 3. Current term is 4. Ops: []
I20260925 03:04:31.090098 25636 raft_consensus.cc:2781] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260925 03:04:31.092097 24511 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 56387.1us
I20260925 03:04:31.118229 24511 leader_election.cc:302] T b030d84c670b48d5bff337ccc0c77626 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:31.092599 26050 raft_consensus.cc:2781] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader election won for term 4
I20260925 03:04:31.092901 25942 consensus_queue.cc:1035] T 442d4a29b1d743599f4b6b4871e26668 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260925 03:04:31.093528 25885 consensus_queue.cc:260] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:31.076090 25897 raft_consensus.cc:504] T cc2df40db08649c9a75e90000b878559 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:31.106129 25982 raft_consensus.cc:504] T e92f377a79354a6ebb4eec67303c13e3 P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:31.128364 25987 raft_consensus.cc:504] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:31.110525 24464 raft_consensus.cc:2454] T b627ecb4a5384ef392aa695e06599c39 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 4.
I20260925 03:04:31.112927 25819 raft_consensus.cc:482] T 7993f9c3e2a44f408bbf5d1e395e2306 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:31.116303 24436 leader_election.cc:302] T 550118bc1a0641ce9803bdbf43f2069f P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:31.123467 25788 raft_consensus.cc:504] T b1e85248d3b04b979dd080caeeb02fdb P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:31.126574 25637 consensus_queue.cc:1046] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 3.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260925 03:04:31.127678 25897 leader_election.cc:288] T cc2df40db08649c9a75e90000b878559 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:31.130625 24473 raft_consensus.cc:1261] T 1414dcfe53f64a78ae413531eb8ec117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260925 03:04:31.138443 24588 leader_election.cc:302] T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:31.145030 24470 raft_consensus.cc:2454] T ceab6b0fccd54e909a64f66fe81a9a1f P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 4.
I20260925 03:04:31.084095 25950 consensus_queue.cc:1035] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.046s
I20260925 03:04:31.122095 25636 raft_consensus.cc:482] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:31.150930 24511 leader_election.cc:398] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 pre-election: Vote denied by peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105) with higher term. Message: Invalid argument: T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 3. Current term is 4.
I20260925 03:04:31.151249 24511 leader_election.cc:401] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20260925 03:04:31.123692 25854 raft_consensus.cc:3037] T b030d84c670b48d5bff337ccc0c77626 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:31.125495 25925 raft_consensus.cc:686] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P eeab93abfdb141a895002ec9f312296c [term 4 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:31.129663 25982 leader_election.cc:288] T e92f377a79354a6ebb4eec67303c13e3 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 3 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:31.146090 25819 raft_consensus.cc:504] T 7993f9c3e2a44f408bbf5d1e395e2306 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:31.157969 25788 leader_election.cc:288] T b1e85248d3b04b979dd080caeeb02fdb P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:31.146220 26050 raft_consensus.cc:686] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [term 4 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:31.123282 25973 consensus_queue.cc:1035] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.032s
I20260925 03:04:31.170248 24618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e92f377a79354a6ebb4eec67303c13e3" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:31.171089 24618 raft_consensus.cc:3037] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:31.150537 25953 raft_consensus.cc:2781] T 550118bc1a0641ce9803bdbf43f2069f P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 03:04:31.153177 25636 raft_consensus.cc:3037] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:31.165585 24615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc2df40db08649c9a75e90000b878559" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:31.174147 24511 leader_election.cc:398] T 0be4e0aa984f4f2b81a58e382e434ad5 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Vote denied by peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105) with higher term. Message: Invalid argument: T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 2. Current term is 3.
I20260925 03:04:31.175071 24615 raft_consensus.cc:3037] T cc2df40db08649c9a75e90000b878559 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:31.175247 24511 leader_election.cc:401] T 0be4e0aa984f4f2b81a58e382e434ad5 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
W20260925 03:04:31.233106 25165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 86304.5us
W20260925 03:04:31.233951 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 68434.5us
I20260925 03:04:31.165719 25799 raft_consensus.cc:2726] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105) with higher term. Message: Invalid argument: T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 3. Current term is 4.
W20260925 03:04:31.240942 24511 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 75822.8us
I20260925 03:04:31.242800 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1e85248d3b04b979dd080caeeb02fdb" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:31.243587 24470 raft_consensus.cc:2454] T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 2.
W20260925 03:04:31.244210 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 105824us
I20260925 03:04:31.245106 24588 leader_election.cc:302] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:31.245661 24614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1e85248d3b04b979dd080caeeb02fdb" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:31.246668 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e54ca1757ee4e42956a9cebbeca4f62" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:31.247265 24464 raft_consensus.cc:3037] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:31.257337 24511 leader_election.cc:302] T ac9357d144aa49789faaf31fe5e6bf89 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:31.269621 24614 raft_consensus.cc:2379] T b1e85248d3b04b979dd080caeeb02fdb P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1bc1f2e62ab44af995165bd5116a9d43 in current term 2: Already voted for candidate eeab93abfdb141a895002ec9f312296c in this term.
I20260925 03:04:31.269424 25165 leader_election.cc:302] T b1e85248d3b04b979dd080caeeb02fdb P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5; no voters:
I20260925 03:04:31.245865 25890 raft_consensus.cc:2726] T 0be4e0aa984f4f2b81a58e382e434ad5 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105) with higher term. Message: Invalid argument: T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 2. Current term is 3.
I20260925 03:04:31.239243 25953 raft_consensus.cc:686] T 550118bc1a0641ce9803bdbf43f2069f P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 LEADER]: Becoming Leader. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:04:31.251525 25960 raft_consensus.cc:482] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:31.269248 25979 consensus_queue.cc:1035] T 48ca6ccd98194cf2af7b5278d15bf4ea P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.005s
I20260925 03:04:31.275870 25865 raft_consensus.cc:979] T a7aa47c48404465b94a36fd819fe3c7a P 869b0a8575b54d08ae2817fad5dd42d5: : Instructing follower eeab93abfdb141a895002ec9f312296c to start an election
I20260925 03:04:31.278992 24511 leader_election.cc:302] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:31.289608 24618 raft_consensus.cc:2454] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 3.
W20260925 03:04:31.292515 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 52564.8us
I20260925 03:04:31.294632 25964 raft_consensus.cc:2781] T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader election won for term 4
I20260925 03:04:31.295236 25964 raft_consensus.cc:686] T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c [term 4 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:31.295583 25819 leader_election.cc:288] T 7993f9c3e2a44f408bbf5d1e395e2306 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:31.272569 25925 consensus_queue.cc:237] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:31.296165 25964 consensus_queue.cc:237] T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:04:31.297247 26059 log.cc:924] Time spent T 0b5df303cd6a4d59aca5c18bbbff24ed P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.054s user 0.000s sys 0.000s
I20260925 03:04:31.297982 25162 leader_election.cc:302] T e92f377a79354a6ebb4eec67303c13e3 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:31.280948 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b627ecb4a5384ef392aa695e06599c39" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:31.301481 25851 raft_consensus.cc:2781] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader election won for term 4
I20260925 03:04:31.302067 25851 raft_consensus.cc:686] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 LEADER]: Becoming Leader. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:04:31.303020 25851 consensus_queue.cc:237] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:31.303900 24464 raft_consensus.cc:2454] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 5.
I20260925 03:04:31.276597 25865 raft_consensus.cc:1067] T a7aa47c48404465b94a36fd819fe3c7a P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 LEADER]: Signalling peer eeab93abfdb141a895002ec9f312296c to start an election
I20260925 03:04:31.271052 25793 raft_consensus.cc:2726] T ac9357d144aa49789faaf31fe5e6bf89 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 03:04:31.274222 25960 raft_consensus.cc:504] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:31.272807 25987 leader_election.cc:288] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 5 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:31.320564 24618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e54ca1757ee4e42956a9cebbeca4f62" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:31.330269 24618 raft_consensus.cc:3037] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:31.306931 25165 leader_election.cc:302] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5; no voters:
I20260925 03:04:31.324918 24615 raft_consensus.cc:2454] T cc2df40db08649c9a75e90000b878559 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 3.
I20260925 03:04:31.291344 24541 raft_consensus.cc:1261] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260925 03:04:31.358572 24546 raft_consensus.cc:1261] T 1414dcfe53f64a78ae413531eb8ec117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260925 03:04:31.305127 26050 consensus_queue.cc:237] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:04:31.358709 25165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 51863.3us
I20260925 03:04:31.357817 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7993f9c3e2a44f408bbf5d1e395e2306" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:31.364337 24547 raft_consensus.cc:2454] T 7993f9c3e2a44f408bbf5d1e395e2306 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 2.
I20260925 03:04:31.362192 24436 leader_election.cc:302] T cc2df40db08649c9a75e90000b878559 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:31.366210 25897 raft_consensus.cc:2781] T cc2df40db08649c9a75e90000b878559 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 03:04:31.366714 25897 raft_consensus.cc:686] T cc2df40db08649c9a75e90000b878559 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 LEADER]: Becoming Leader. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:04:31.367604 25897 consensus_queue.cc:237] T cc2df40db08649c9a75e90000b878559 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:31.368191 25854 raft_consensus.cc:2726] T b030d84c670b48d5bff337ccc0c77626 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260925 03:04:31.318488 25973 raft_consensus.cc:2781] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader election won for term 4
I20260925 03:04:31.376425 24586 leader_election.cc:302] T 7993f9c3e2a44f408bbf5d1e395e2306 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
W20260925 03:04:31.378623 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 133599us
I20260925 03:04:31.355156 25788 raft_consensus.cc:2781] T b1e85248d3b04b979dd080caeeb02fdb P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 03:04:31.292148 25804 consensus_queue.cc:1046] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.081s
I20260925 03:04:31.373605 25197 raft_consensus.cc:3037] T 367277daf8d04778b3fa4ed94d71580d P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:31.373934 25987 raft_consensus.cc:2781] T e92f377a79354a6ebb4eec67303c13e3 P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 03:04:31.379755 26065 raft_consensus.cc:482] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:31.396538 24437 leader_election.cc:302] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:31.393633 24615 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a7aa47c48404465b94a36fd819fe3c7a"
dest_uuid: "eeab93abfdb141a895002ec9f312296c"
from {username='slave'} at 127.0.0.1:38570
I20260925 03:04:31.397981 24615 raft_consensus.cc:482] T a7aa47c48404465b94a36fd819fe3c7a P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 03:04:31.398325 24615 raft_consensus.cc:3037] T a7aa47c48404465b94a36fd819fe3c7a P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:31.382849 25788 raft_consensus.cc:482] T b1e85248d3b04b979dd080caeeb02fdb P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:31.375322 25982 consensus_queue.cc:1035] T 04e068a0f80941dc9138cee063f53bab P 1bc1f2e62ab44af995165bd5116a9d43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.019s
I20260925 03:04:31.389112 25984 raft_consensus.cc:2726] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260925 03:04:31.409641 24618 raft_consensus.cc:2454] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 5.
I20260925 03:04:31.385701 25973 raft_consensus.cc:686] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [term 4 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:31.403630 25788 raft_consensus.cc:3037] T b1e85248d3b04b979dd080caeeb02fdb P 1bc1f2e62ab44af995165bd5116a9d43 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:31.427268 24615 raft_consensus.cc:504] T a7aa47c48404465b94a36fd819fe3c7a P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:04:31.434314 25163 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 129403us
W20260925 03:04:31.474236 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 94667.7us
I20260925 03:04:31.476593 24615 leader_election.cc:288] T a7aa47c48404465b94a36fd819fe3c7a P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
W20260925 03:04:31.413888 25162 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 115998us
W20260925 03:04:31.483639 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 107301us
I20260925 03:04:31.415474 26067 consensus_queue.cc:1035] T 1414dcfe53f64a78ae413531eb8ec117 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.066s
I20260925 03:04:31.489248 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7993f9c3e2a44f408bbf5d1e395e2306" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
W20260925 03:04:31.489969 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 93523.1us
I20260925 03:04:31.493167 24467 raft_consensus.cc:1261] T 0be4e0aa984f4f2b81a58e382e434ad5 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260925 03:04:31.391415 25953 consensus_queue.cc:237] T 550118bc1a0641ce9803bdbf43f2069f P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:04:31.481915 26062 log.cc:924] Time spent T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5: Append to log took a long time: real 0.086s user 0.001s sys 0.000s
I20260925 03:04:31.495268 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7aa47c48404465b94a36fd819fe3c7a" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:31.436208 26076 raft_consensus.cc:2781] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [term 5 FOLLOWER]: Leader election won for term 5
I20260925 03:04:31.434751 25957 raft_consensus.cc:2726] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260925 03:04:31.476948 25987 raft_consensus.cc:686] T e92f377a79354a6ebb4eec67303c13e3 P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 LEADER]: Becoming Leader. State: Replica: 1bc1f2e62ab44af995165bd5116a9d43, State: Running, Role: LEADER
I20260925 03:04:31.505838 25960 leader_election.cc:288] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:31.507825 24615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b030d84c670b48d5bff337ccc0c77626" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:31.508446 24541 raft_consensus.cc:1261] T 367277daf8d04778b3fa4ed94d71580d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260925 03:04:31.509496 26075 consensus_queue.cc:1035] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260925 03:04:31.508491 24615 raft_consensus.cc:2454] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 4.
I20260925 03:04:31.505883 26070 raft_consensus.cc:2781] T 7993f9c3e2a44f408bbf5d1e395e2306 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260925 03:04:31.511400 26070 raft_consensus.cc:482] T 7993f9c3e2a44f408bbf5d1e395e2306 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:31.511678 24436 leader_election.cc:302] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:31.511713 26070 raft_consensus.cc:3037] T 7993f9c3e2a44f408bbf5d1e395e2306 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:31.513401 26071 consensus_queue.cc:1035] T 1414dcfe53f64a78ae413531eb8ec117 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.020s
I20260925 03:04:31.514022 25938 raft_consensus.cc:2781] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260925 03:04:31.514420 25938 raft_consensus.cc:482] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:31.514724 25938 raft_consensus.cc:3037] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:31.556315 24615 raft_consensus.cc:1226] T a7aa47c48404465b94a36fd819fe3c7a P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 1. Current term is 2. Ops: []
I20260925 03:04:31.545385 24473 raft_consensus.cc:2454] T 7993f9c3e2a44f408bbf5d1e395e2306 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 1.
W20260925 03:04:31.554085 24511 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 273883us
I20260925 03:04:31.437166 25973 consensus_queue.cc:237] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:31.420841 26065 raft_consensus.cc:504] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:31.492349 25804 raft_consensus.cc:482] T a7aa47c48404465b94a36fd819fe3c7a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:31.560367 24437 leader_election.cc:302] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:31.581810 25788 raft_consensus.cc:504] T b1e85248d3b04b979dd080caeeb02fdb P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:31.582661 26068 consensus_queue.cc:1035] T 367277daf8d04778b3fa4ed94d71580d P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.017s
I20260925 03:04:31.584079 25788 leader_election.cc:288] T b1e85248d3b04b979dd080caeeb02fdb P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 2 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:31.575434 25741 raft_consensus.cc:2726] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260925 03:04:31.570493 25162 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 50982.9us
I20260925 03:04:31.572098 26076 raft_consensus.cc:686] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [term 5 LEADER]: Becoming Leader. State: Replica: 1bc1f2e62ab44af995165bd5116a9d43, State: Running, Role: LEADER
I20260925 03:04:31.578693 24465 raft_consensus.cc:1261] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 4 index: 4. (term mismatch)
I20260925 03:04:31.601024 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1e85248d3b04b979dd080caeeb02fdb" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:31.567843 25957 consensus_queue.cc:1046] T a7aa47c48404465b94a36fd819fe3c7a P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.029s
I20260925 03:04:31.562952 26081 consensus_queue.cc:1035] T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.043s
I20260925 03:04:31.572072 25804 raft_consensus.cc:504] T a7aa47c48404465b94a36fd819fe3c7a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:31.608081 24437 leader_election.cc:302] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:31.596280 24541 raft_consensus.cc:1261] T 0be4e0aa984f4f2b81a58e382e434ad5 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260925 03:04:31.609090 24465 pending_rounds.cc:76] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5: Aborting all ops after (but not including) 3
I20260925 03:04:31.601768 25788 consensus_queue.cc:1035] T e6b6ec010caf4689b8c90449764f5cf9 P 1bc1f2e62ab44af995165bd5116a9d43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.025s
I20260925 03:04:31.636458 24547 raft_consensus.cc:3037] T a7aa47c48404465b94a36fd819fe3c7a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:31.638504 26074 raft_consensus.cc:482] T 7e49248643df4131a1c42e74244e05c2 P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:31.638661 25741 raft_consensus.cc:2726] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260925 03:04:31.639380 25192 raft_consensus.cc:1261] T a4abe0ea757e4886be4945c5001e529a P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260925 03:04:31.613637 25925 raft_consensus.cc:482] T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:31.641543 25925 raft_consensus.cc:504] T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:31.644411 25925 leader_election.cc:288] T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:31.644503 25799 consensus_queue.cc:1046] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260925 03:04:31.645815 25885 raft_consensus.cc:3037] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:31.646972 25854 raft_consensus.cc:482] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:31.647486 25854 raft_consensus.cc:504] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:31.648093 25192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:31.651592 26078 raft_consensus.cc:3032] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 LEADER]: Stepping down as leader of term 3
I20260925 03:04:31.652103 26078 raft_consensus.cc:729] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:04:31.653059 26078 consensus_queue.cc:260] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:31.654563 26076 consensus_queue.cc:237] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:31.634164 24465 pending_rounds.cc:97] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5: Aborting uncommitted CHANGE_CONFIG_OP operation due to leader change: 3.4
I20260925 03:04:31.669016 24465 raft_consensus.cc:2894] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Aborting config change with OpId 3.4: Aborted: Op aborted by new leader
I20260925 03:04:31.629676 26071 raft_consensus.cc:482] T 100c113a9fd84cc8adf2ec7b33ea0093 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:31.669816 26074 raft_consensus.cc:504] T 7e49248643df4131a1c42e74244e05c2 P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:31.664234 25783 raft_consensus.cc:482] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:31.670977 25783 raft_consensus.cc:504] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:31.672909 25783 leader_election.cc:288] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:31.675822 25938 raft_consensus.cc:504] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:31.614303 25987 consensus_queue.cc:237] T e92f377a79354a6ebb4eec67303c13e3 P 1bc1f2e62ab44af995165bd5116a9d43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:31.667145 26093 consensus_queue.cc:1035] T a4abe0ea757e4886be4945c5001e529a P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260925 03:04:31.668361 24615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1e85248d3b04b979dd080caeeb02fdb" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:31.670010 26071 raft_consensus.cc:504] T 100c113a9fd84cc8adf2ec7b33ea0093 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:31.664268 24467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
W20260925 03:04:31.669435 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 65471.2us
I20260925 03:04:31.735488 24547 raft_consensus.cc:2454] T a7aa47c48404465b94a36fd819fe3c7a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 2.
I20260925 03:04:31.676584 24468 raft_consensus.cc:2454] T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 2.
I20260925 03:04:31.736977 24615 raft_consensus.cc:2379] T b1e85248d3b04b979dd080caeeb02fdb P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1bc1f2e62ab44af995165bd5116a9d43 in current term 2: Already voted for candidate eeab93abfdb141a895002ec9f312296c in this term.
I20260925 03:04:31.664085 25804 leader_election.cc:288] T a7aa47c48404465b94a36fd819fe3c7a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:31.750021 24467 raft_consensus.cc:2454] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 4.
I20260925 03:04:31.782646 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7691c6b475428c82d6181d693d26f3" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:31.783178 25165 leader_election.cc:302] T b1e85248d3b04b979dd080caeeb02fdb P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5; no voters:
I20260925 03:04:31.783649 24618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:31.784276 24618 raft_consensus.cc:2454] T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 3.
I20260925 03:04:31.762018 24614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f68804eb5a244d87bb395d50683c6012" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:31.683053 25197 raft_consensus.cc:1261] T 367277daf8d04778b3fa4ed94d71580d P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 4 index: 4. (term mismatch)
I20260925 03:04:31.669509 24465 raft_consensus.cc:2866] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted by new leader
I20260925 03:04:31.738847 25783 raft_consensus.cc:482] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:31.691511 25854 leader_election.cc:288] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
W20260925 03:04:31.798910 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 49938.5us
I20260925 03:04:31.677510 26065 leader_election.cc:288] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:31.794641 25885 raft_consensus.cc:482] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:31.794210 25982 raft_consensus.cc:2781] T b1e85248d3b04b979dd080caeeb02fdb P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 03:04:31.798163 25192 raft_consensus.cc:2379] T 7e49248643df4131a1c42e74244e05c2 P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c in current term 4: Already voted for candidate 1bc1f2e62ab44af995165bd5116a9d43 in this term.
I20260925 03:04:31.819126 24437 leader_election.cc:302] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:31.682231 26074 leader_election.cc:288] T 7e49248643df4131a1c42e74244e05c2 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:31.690985 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:31.681676 26070 raft_consensus.cc:504] T 7993f9c3e2a44f408bbf5d1e395e2306 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:31.796200 25197 pending_rounds.cc:76] T 367277daf8d04778b3fa4ed94d71580d P 1bc1f2e62ab44af995165bd5116a9d43: Aborting all ops after (but not including) 3
I20260925 03:04:31.802359 26084 raft_consensus.cc:482] T c88cbd39e01f44628a094af516be2833 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:31.816788 24613 raft_consensus.cc:1261] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260925 03:04:31.817508 25885 raft_consensus.cc:504] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:31.684906 25636 raft_consensus.cc:504] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:31.843158 24471 raft_consensus.cc:2454] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 4.
I20260925 03:04:31.854475 24588 leader_election.cc:302] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:31.830997 26075 raft_consensus.cc:482] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260925 03:04:31.835215 24511 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 271710us
I20260925 03:04:31.839183 25197 pending_rounds.cc:97] T 367277daf8d04778b3fa4ed94d71580d P 1bc1f2e62ab44af995165bd5116a9d43: Aborting uncommitted CHANGE_CONFIG_OP operation due to leader change: 3.4
I20260925 03:04:31.836486 25982 raft_consensus.cc:686] T b1e85248d3b04b979dd080caeeb02fdb P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 LEADER]: Becoming Leader. State: Replica: 1bc1f2e62ab44af995165bd5116a9d43, State: Running, Role: LEADER
I20260925 03:04:31.842113 25957 raft_consensus.cc:2726] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260925 03:04:31.806068 25783 raft_consensus.cc:504] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:31.843775 26093 raft_consensus.cc:482] T b1e85248d3b04b979dd080caeeb02fdb P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:31.887213 25964 raft_consensus.cc:2781] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260925 03:04:31.887825 25964 raft_consensus.cc:482] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:31.888226 25964 raft_consensus.cc:3037] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:31.892307 25804 raft_consensus.cc:482] T c85f5d1d3f004cf5817bb594fd8e6469 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:31.892967 25804 raft_consensus.cc:504] T c85f5d1d3f004cf5817bb594fd8e6469 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:31.894043 25942 raft_consensus.cc:3032] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [term 3 LEADER]: Stepping down as leader of term 3
I20260925 03:04:31.894423 25942 raft_consensus.cc:729] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:31.894959 25804 leader_election.cc:288] T c85f5d1d3f004cf5817bb594fd8e6469 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:31.896315 24465 raft_consensus.cc:1261] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260925 03:04:31.897140 26090 log.cc:924] Time spent T abd5b8dc1b7e49e3aa9af900d73ff0e7 P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.054s user 0.001s sys 0.000s
I20260925 03:04:31.898038 26094 consensus_queue.cc:1046] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260925 03:04:31.899163 26094 raft_consensus.cc:482] T 8e5819c15eac4544a57e5ee6c9d4f117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:31.896584 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ac466793d094325ace224708b2158ed" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:31.841636 24466 raft_consensus.cc:2454] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 4.
I20260925 03:04:31.900175 24548 raft_consensus.cc:2454] T 8ac466793d094325ace224708b2158ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 3.
I20260925 03:04:31.904209 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:31.905730 24543 raft_consensus.cc:2379] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c in current term 4: Already voted for candidate 1bc1f2e62ab44af995165bd5116a9d43 in this term.
W20260925 03:04:31.909775 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 90747.1us
I20260925 03:04:31.877797 25197 raft_consensus.cc:2894] T 367277daf8d04778b3fa4ed94d71580d P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Aborting config change with OpId 3.4: Aborted: Op aborted by new leader
I20260925 03:04:31.892544 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ac466793d094325ace224708b2158ed" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:31.914281 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e92f377a79354a6ebb4eec67303c13e3" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:31.917920 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4abe0ea757e4886be4945c5001e529a" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:31.895215 25942 consensus_queue.cc:260] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
W20260925 03:04:31.918725 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 78439us
I20260925 03:04:31.913352 25783 leader_election.cc:288] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:31.927865 25983 consensus_queue.cc:1035] T 367277daf8d04778b3fa4ed94d71580d P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.010s
I20260925 03:04:31.903890 26093 raft_consensus.cc:504] T b1e85248d3b04b979dd080caeeb02fdb P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:31.921207 25973 consensus_queue.cc:1035] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.012s
I20260925 03:04:31.932780 26086 raft_consensus.cc:3032] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 LEADER]: Stepping down as leader of term 3
I20260925 03:04:31.932369 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
W20260925 03:04:31.933107 26087 log.cc:924] Time spent T cc2df40db08649c9a75e90000b878559 P 869b0a8575b54d08ae2817fad5dd42d5: Append to log took a long time: real 0.088s user 0.001s sys 0.000s
I20260925 03:04:31.933163 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:31.864874 26084 raft_consensus.cc:504] T c88cbd39e01f44628a094af516be2833 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:31.908937 25804 consensus_queue.cc:1046] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.022s
I20260925 03:04:31.934432 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "100c113a9fd84cc8adf2ec7b33ea0093" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:31.921738 25197 raft_consensus.cc:2866] T 367277daf8d04778b3fa4ed94d71580d P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 LEARNER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted by new leader
I20260925 03:04:31.935144 24437 leader_election.cc:302] T 8e5819c15eac4544a57e5ee6c9d4f117 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:31.877298 26075 raft_consensus.cc:504] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:31.929171 25163 leader_election.cc:302] T a4abe0ea757e4886be4945c5001e529a P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:31.939978 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e92f377a79354a6ebb4eec67303c13e3" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:31.949002 24543 raft_consensus.cc:2362] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 4. Re-sending same reply.
W20260925 03:04:31.977738 24511 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 85845.5us
I20260925 03:04:31.968590 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7aa47c48404465b94a36fd819fe3c7a" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:31.995256 24470 raft_consensus.cc:3032] T a7aa47c48404465b94a36fd819fe3c7a P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 LEADER]: Stepping down as leader of term 1
I20260925 03:04:31.995591 24470 raft_consensus.cc:729] T a7aa47c48404465b94a36fd819fe3c7a P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:04:31.912197 26094 raft_consensus.cc:504] T 8e5819c15eac4544a57e5ee6c9d4f117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:32.091198 23300 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260925 03:04:31.943729 25938 leader_election.cc:288] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:32.127987 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7691c6b475428c82d6181d693d26f3" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:32.128857 24542 raft_consensus.cc:2454] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 4.
I20260925 03:04:31.950719 26086 raft_consensus.cc:729] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:04:31.965636 26071 leader_election.cc:288] T 100c113a9fd84cc8adf2ec7b33ea0093 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
W20260925 03:04:32.142062 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 213932us
W20260925 03:04:32.154194 25163 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 225098us
I20260925 03:04:32.116649 24470 consensus_queue.cc:260] T a7aa47c48404465b94a36fd819fe3c7a P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:32.174167 24542 raft_consensus.cc:1261] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260925 03:04:31.957398 25964 raft_consensus.cc:504] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:32.123584 25783 raft_consensus.cc:2726] T 8e5819c15eac4544a57e5ee6c9d4f117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260925 03:04:32.124055 24466 raft_consensus.cc:2454] T 100c113a9fd84cc8adf2ec7b33ea0093 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 2.
I20260925 03:04:32.190407 26093 leader_election.cc:288] T b1e85248d3b04b979dd080caeeb02fdb P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:32.190747 25953 consensus_queue.cc:1046] T 21c43af5e6844fdaacc8c625fa01fd8d P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260925 03:04:32.191926 25953 raft_consensus.cc:3032] T 21c43af5e6844fdaacc8c625fa01fd8d P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 LEADER]: Stepping down as leader of term 1
I20260925 03:04:32.192305 25953 raft_consensus.cc:729] T 21c43af5e6844fdaacc8c625fa01fd8d P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:04:32.193122 25953 consensus_queue.cc:260] T 21c43af5e6844fdaacc8c625fa01fd8d P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:32.194434 25953 raft_consensus.cc:3037] T 21c43af5e6844fdaacc8c625fa01fd8d P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Advancing to term 2
W20260925 03:04:32.195266 26099 log.cc:924] Time spent T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.078s user 0.000s sys 0.000s
I20260925 03:04:32.135104 26076 raft_consensus.cc:2726] T a4abe0ea757e4886be4945c5001e529a P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 03:04:32.190323 24437 leader_election.cc:302] T 9e8af2641e124f9589c5ccfdac785f63 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:32.188378 25804 raft_consensus.cc:482] T 420163a6cdce412ebc21c25fe3835638 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:32.154369 24465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7993f9c3e2a44f408bbf5d1e395e2306" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
W20260925 03:04:32.217273 25163 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 61431.5us
I20260925 03:04:32.090526 25982 consensus_queue.cc:237] T b1e85248d3b04b979dd080caeeb02fdb P 1bc1f2e62ab44af995165bd5116a9d43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:32.163311 25885 leader_election.cc:288] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:32.185331 26070 leader_election.cc:288] T 7993f9c3e2a44f408bbf5d1e395e2306 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 3 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
W20260925 03:04:32.234786 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 44573.1us
I20260925 03:04:32.199416 25897 raft_consensus.cc:482] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:32.241223 25197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "293ef69caf514ac98510a68495f54eda" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43"
I20260925 03:04:32.242187 24539 raft_consensus.cc:2454] T 8ac466793d094325ace224708b2158ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 3.
I20260925 03:04:32.216532 25804 raft_consensus.cc:504] T 420163a6cdce412ebc21c25fe3835638 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:32.209385 26118 raft_consensus.cc:3032] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 LEADER]: Stepping down as leader of term 3
I20260925 03:04:32.244839 26118 raft_consensus.cc:729] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:04:32.205587 25636 leader_election.cc:288] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:32.245729 26118 consensus_queue.cc:260] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:32.246958 26118 raft_consensus.cc:3037] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Advancing to term 4
W20260925 03:04:32.248245 26102 log.cc:924] Time spent T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.128s user 0.001s sys 0.000s
I20260925 03:04:32.199677 25983 consensus_queue.cc:1035] T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.065s
I20260925 03:04:32.214659 25783 raft_consensus.cc:2726] T 9e8af2641e124f9589c5ccfdac785f63 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260925 03:04:32.237061 24465 raft_consensus.cc:3037] T 7993f9c3e2a44f408bbf5d1e395e2306 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Advancing to term 3
I20260925 03:04:32.283461 24437 leader_election.cc:302] T 293ef69caf514ac98510a68495f54eda P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: 1bc1f2e62ab44af995165bd5116a9d43, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:32.281543 26084 leader_election.cc:288] T c88cbd39e01f44628a094af516be2833 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
W20260925 03:04:32.295429 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 325924us
I20260925 03:04:32.259379 25897 raft_consensus.cc:504] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:32.289156 25804 leader_election.cc:288] T 420163a6cdce412ebc21c25fe3835638 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:32.333527 24621 raft_consensus.cc:1226] T 5278a60f97da4e10aecd50a8d9a9a43c P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 1. Current term is 2. Ops: []
I20260925 03:04:32.297932 25783 raft_consensus.cc:2726] T 293ef69caf514ac98510a68495f54eda P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260925 03:04:32.371433 24615 raft_consensus.cc:1226] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 2. Current term is 3. Ops: []
I20260925 03:04:32.375258 25198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1e85248d3b04b979dd080caeeb02fdb" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:32.379755 24619 raft_consensus.cc:1226] T 37a157996e944db2906110d87ecb0c54 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 3. Current term is 4. Ops: []
I20260925 03:04:32.373426 24437 leader_election.cc:302] T 11bb46bce3564841b7674e6a693bd2e6 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:32.383571 24614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7691c6b475428c82d6181d693d26f3" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:32.379666 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7aa47c48404465b94a36fd819fe3c7a" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:32.384359 24614 raft_consensus.cc:2379] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 5: Already voted for candidate eeab93abfdb141a895002ec9f312296c in this term.
I20260925 03:04:32.322899 25190 raft_consensus.cc:1226] T e92f377a79354a6ebb4eec67303c13e3 P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 LEADER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 2. Current term is 3. Ops: []
I20260925 03:04:32.384765 24588 leader_election.cc:302] T 100c113a9fd84cc8adf2ec7b33ea0093 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:32.383688 24622 raft_consensus.cc:1226] T a4abe0ea757e4886be4945c5001e529a P eeab93abfdb141a895002ec9f312296c [term 2 LEADER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 1. Current term is 2. Ops: []
I20260925 03:04:32.438405 25163 leader_election.cc:302] T 8ac466793d094325ace224708b2158ed P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:04:32.440677 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1e85248d3b04b979dd080caeeb02fdb" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:32.441320 24468 raft_consensus.cc:1261] T 3d89396576884f7c8901dba54f783947 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20260925 03:04:32.441967 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 57300.6us
I20260925 03:04:32.441366 24471 raft_consensus.cc:2454] T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 2.
I20260925 03:04:32.444618 24618 raft_consensus.cc:1226] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [term 4 LEADER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 3. Current term is 4. Ops: []
W20260925 03:04:32.445956 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 118104us
I20260925 03:04:32.446977 24615 raft_consensus.cc:1226] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P eeab93abfdb141a895002ec9f312296c [term 4 LEADER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 3. Current term is 4. Ops: []
I20260925 03:04:32.236268 26086 consensus_queue.cc:260] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:32.379405 26105 raft_consensus.cc:482] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:32.450279 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "420163a6cdce412ebc21c25fe3835638" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:32.374702 25191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5aec38e66a947c681e6c2a72b05e6ca" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43"
I20260925 03:04:32.462154 24619 raft_consensus.cc:2379] T 420163a6cdce412ebc21c25fe3835638 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 4: Already voted for candidate eeab93abfdb141a895002ec9f312296c in this term.
I20260925 03:04:32.385200 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5332a5d329493991dff994e70ea92c" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:32.327679 25983 consensus_queue.cc:1035] T 367277daf8d04778b3fa4ed94d71580d P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.122s
I20260925 03:04:32.489861 24588 leader_election.cc:302] T b1e85248d3b04b979dd080caeeb02fdb P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters: 1bc1f2e62ab44af995165bd5116a9d43
I20260925 03:04:32.445200 26075 leader_election.cc:288] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:32.444540 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5aec38e66a947c681e6c2a72b05e6ca" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:32.463078 24616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c85f5d1d3f004cf5817bb594fd8e6469" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:32.454114 25804 raft_consensus.cc:482] T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:32.456300 26094 leader_election.cc:288] T 8e5819c15eac4544a57e5ee6c9d4f117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:32.460120 25783 raft_consensus.cc:2726] T 11bb46bce3564841b7674e6a693bd2e6 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260925 03:04:32.477823 25191 raft_consensus.cc:3037] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:32.546864 24468 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0be4e0aa984f4f2b81a58e382e434ad5"
dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
mode: GRACEFUL
new_leader_uuid: "eeab93abfdb141a895002ec9f312296c"
from {username='slave'} at 127.0.0.1:38832
I20260925 03:04:32.553759 24468 raft_consensus.cc:595] T 0be4e0aa984f4f2b81a58e382e434ad5 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Received request to transfer leadership to TS eeab93abfdb141a895002ec9f312296c
I20260925 03:04:32.554131 24468 raft_consensus.cc:603] T 0be4e0aa984f4f2b81a58e382e434ad5 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260925 03:04:32.558938 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c88cbd39e01f44628a094af516be2833" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:32.468888 25964 leader_election.cc:288] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
W20260925 03:04:32.534883 26111 log.cc:924] Time spent T 48ca6ccd98194cf2af7b5278d15bf4ea P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.059s user 0.001s sys 0.000s
I20260925 03:04:32.498183 26107 raft_consensus.cc:482] T c88cbd39e01f44628a094af516be2833 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:32.478609 25741 raft_consensus.cc:482] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:32.536427 24466 raft_consensus.cc:3037] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:32.536855 26086 raft_consensus.cc:3037] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Advancing to term 4
W20260925 03:04:32.449478 25162 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 67284.3us
I20260925 03:04:32.543668 25973 raft_consensus.cc:2781] T b1e85248d3b04b979dd080caeeb02fdb P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260925 03:04:32.594288 24471 raft_consensus.cc:2454] T c88cbd39e01f44628a094af516be2833 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 3.
I20260925 03:04:32.537220 24469 raft_consensus.cc:2454] T 4e5332a5d329493991dff994e70ea92c P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 4.
I20260925 03:04:32.603544 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e5819c15eac4544a57e5ee6c9d4f117" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:32.607563 24468 raft_consensus.cc:2454] T 8e5819c15eac4544a57e5ee6c9d4f117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 2.
W20260925 03:04:32.569062 26114 log.cc:924] Time spent T 1414dcfe53f64a78ae413531eb8ec117 P 869b0a8575b54d08ae2817fad5dd42d5: Append to log took a long time: real 0.125s user 0.001s sys 0.000s
I20260925 03:04:32.606302 24588 leader_election.cc:302] T c88cbd39e01f44628a094af516be2833 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:32.610138 24588 leader_election.cc:302] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:32.568339 25783 raft_consensus.cc:482] T 442d4a29b1d743599f4b6b4871e26668 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:32.614984 24437 leader_election.cc:302] T 7993f9c3e2a44f408bbf5d1e395e2306 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:04:32.617412 24588 leader_election.cc:302] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:32.546056 25804 raft_consensus.cc:504] T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:32.619199 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7691c6b475428c82d6181d693d26f3" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:32.624418 24471 raft_consensus.cc:3037] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:32.565763 26094 consensus_queue.cc:1046] T 5278a60f97da4e10aecd50a8d9a9a43c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.051s
I20260925 03:04:32.605450 25973 raft_consensus.cc:482] T b1e85248d3b04b979dd080caeeb02fdb P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:32.624042 24617 raft_consensus.cc:1226] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 3. Current term is 4. Ops: []
I20260925 03:04:32.468417 26105 raft_consensus.cc:504] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:32.618341 25983 raft_consensus.cc:2781] T c88cbd39e01f44628a094af516be2833 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260925 03:04:32.625697 26068 raft_consensus.cc:2781] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260925 03:04:32.602807 26124 consensus_queue.cc:1046] T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.017s
I20260925 03:04:32.627589 25897 leader_election.cc:288] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
W20260925 03:04:32.628845 25163 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 190525us
I20260925 03:04:32.625207 25987 raft_consensus.cc:482] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
W20260925 03:04:32.649269 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 117953us
W20260925 03:04:32.631422 24511 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 166703us
I20260925 03:04:32.633642 25973 raft_consensus.cc:3037] T b1e85248d3b04b979dd080caeeb02fdb P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:32.609833 25741 raft_consensus.cc:504] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:32.658633 24471 raft_consensus.cc:2454] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 5.
I20260925 03:04:32.634596 26126 consensus_queue.cc:1046] T 37a157996e944db2906110d87ecb0c54 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.004s
I20260925 03:04:32.662204 24588 leader_election.cc:302] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
W20260925 03:04:32.669503 24586 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.511s user 0.010s sys 0.006s
W20260925 03:04:32.670003 24586 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.511s user 0.010s sys 0.007s
I20260925 03:04:32.644964 25983 raft_consensus.cc:482] T c88cbd39e01f44628a094af516be2833 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:32.647585 26068 raft_consensus.cc:482] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:32.646279 26120 raft_consensus.cc:2781] T 100c113a9fd84cc8adf2ec7b33ea0093 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260925 03:04:32.678560 24473 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 713) took 1072 ms. Trace:
I20260925 03:04:32.693337 24473 rpcz_store.cc:276] 0925 03:04:31.598487 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:31.598633 (+ 146us) service_pool.cc:227] Handling call
0925 03:04:31.608502 (+ 9869us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:04:31.628290 (+ 19788us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:31.628368 (+ 78us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:04:31.630488 (+ 2120us) log.cc:842] Serialized 26 byte log entry
0925 03:04:32.105101 (+474613us) raft_consensus.cc:1631] Marking committed up to 1
0925 03:04:32.105140 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:32.105213 (+ 73us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:04:32.670491 (+565278us) raft_consensus.cc:1671] finished
0925 03:04:32.670514 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:32.670706 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":14973,"raft.queue_time_us":29168,"raft.run_cpu_time_us":119,"raft.run_wall_time_us":121,"thread_start_us":319858,"threads_started":1,"wal-append.queue_time_us":631742}
I20260925 03:04:32.609508 26107 raft_consensus.cc:504] T c88cbd39e01f44628a094af516be2833 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:32.659544 25897 raft_consensus.cc:482] T 3d89396576884f7c8901dba54f783947 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:32.633490 25783 raft_consensus.cc:504] T 442d4a29b1d743599f4b6b4871e26668 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:04:32.673085 24587 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.589s user 0.014s sys 0.060s
W20260925 03:04:32.705479 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 54306.1us
I20260925 03:04:32.679032 25983 raft_consensus.cc:3037] T c88cbd39e01f44628a094af516be2833 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:32.678625 26104 raft_consensus.cc:2781] T 8ac466793d094325ace224708b2158ed P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 03:04:32.692409 26068 raft_consensus.cc:3037] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:32.692138 26120 raft_consensus.cc:482] T 100c113a9fd84cc8adf2ec7b33ea0093 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260925 03:04:32.709429 24587 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.763s user 0.014s sys 0.060s
I20260925 03:04:32.671592 25987 raft_consensus.cc:504] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
W20260925 03:04:32.721355 26109 log.cc:924] Time spent T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5: Append to log took a long time: real 0.365s user 0.000s sys 0.002s
I20260925 03:04:32.709208 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "100c113a9fd84cc8adf2ec7b33ea0093" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:32.735625 26115 raft_consensus.cc:2781] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Leader election won for term 5
I20260925 03:04:32.736150 26119 raft_consensus.cc:482] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:32.735932 26121 consensus_queue.cc:1046] T 43366a0a2f9d4344bf4e37ffd290ef8a P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.116s
I20260925 03:04:32.739758 26121 raft_consensus.cc:3032] T 43366a0a2f9d4344bf4e37ffd290ef8a P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 LEADER]: Stepping down as leader of term 1
I20260925 03:04:32.740114 26121 raft_consensus.cc:729] T 43366a0a2f9d4344bf4e37ffd290ef8a P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:04:32.740239 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7993f9c3e2a44f408bbf5d1e395e2306" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:32.741062 24541 raft_consensus.cc:3037] T 7993f9c3e2a44f408bbf5d1e395e2306 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:32.673280 26066 consensus_queue.cc:1035] T 3d89396576884f7c8901dba54f783947 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.224s
I20260925 03:04:32.719524 25805 raft_consensus.cc:482] T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1bc1f2e62ab44af995165bd5116a9d43)
I20260925 03:04:32.809813 25793 raft_consensus.cc:3032] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 LEADER]: Stepping down as leader of term 3
I20260925 03:04:32.820086 25984 raft_consensus.cc:482] T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:32.733618 26123 raft_consensus.cc:482] T 420163a6cdce412ebc21c25fe3835638 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:32.821730 26094 consensus_queue.cc:1046] T a4abe0ea757e4886be4945c5001e529a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260925 03:04:32.785020 25195 raft_consensus.cc:1226] T a4abe0ea757e4886be4945c5001e529a P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 1. Current term is 2. Ops: []
I20260925 03:04:32.806509 24437 leader_election.cc:302] T f22ddd7d1c444dc58f45a1fb0b3e02bc P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:32.772658 24473 raft_consensus.cc:1261] T b627ecb4a5384ef392aa695e06599c39 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260925 03:04:32.864637 25741 leader_election.cc:288] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:32.726802 26120 raft_consensus.cc:3037] T 100c113a9fd84cc8adf2ec7b33ea0093 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:32.707163 25897 raft_consensus.cc:504] T 3d89396576884f7c8901dba54f783947 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:32.743669 26104 raft_consensus.cc:686] T 8ac466793d094325ace224708b2158ed P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 LEADER]: Becoming Leader. State: Replica: 1bc1f2e62ab44af995165bd5116a9d43, State: Running, Role: LEADER
W20260925 03:04:32.786720 26108 log.cc:924] Time spent T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43: Append to log took a long time: real 0.260s user 0.001s sys 0.000s
I20260925 03:04:32.822028 26105 leader_election.cc:288] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:32.772387 24464 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 705) took 1293 ms. Trace:
I20260925 03:04:32.807013 26125 raft_consensus.cc:2781] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260925 03:04:32.812036 26071 raft_consensus.cc:482] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:32.867518 25793 raft_consensus.cc:729] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
W20260925 03:04:32.910202 26135 log.cc:924] Time spent T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.096s user 0.000s sys 0.000s
I20260925 03:04:32.910461 25783 leader_election.cc:288] T 442d4a29b1d743599f4b6b4871e26668 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:32.912449 26126 raft_consensus.cc:3032] T 37a157996e944db2906110d87ecb0c54 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 LEADER]: Stepping down as leader of term 3
W20260925 03:04:32.817965 24548 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.564s user 0.012s sys 0.012s
W20260925 03:04:32.958431 24548 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.705s user 0.012s sys 0.012s
I20260925 03:04:32.822991 24539 raft_consensus.cc:2454] T 100c113a9fd84cc8adf2ec7b33ea0093 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 2.
W20260925 03:04:32.914958 25197 consensus_meta.cc:287] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43: Time spent flushing consensus metadata: real 0.673s user 0.010s sys 0.002s
I20260925 03:04:32.747372 26119 raft_consensus.cc:504] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:04:32.966041 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 192936us
W20260925 03:04:32.923732 24511 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 262967us
I20260925 03:04:32.905295 25804 leader_election.cc:288] T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:32.896374 24617 raft_consensus.cc:1261] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Refusing update from remote peer 1bc1f2e62ab44af995165bd5116a9d43: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260925 03:04:32.869074 26121 consensus_queue.cc:260] T 43366a0a2f9d4344bf4e37ffd290ef8a P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:32.938246 25982 raft_consensus.cc:482] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:32.905740 26136 raft_consensus.cc:482] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:32.937119 24464 rpcz_store.cc:276] 0925 03:04:31.478922 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:31.483584 (+ 4662us) service_pool.cc:227] Handling call
0925 03:04:31.897491 (+413907us) spinlock_profiling.cc:243] Waited 17 ms on lock 0x7b5800001598. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f43733ecf11 00007f4373570851 00007f4373570702 00007f437c460b09 00007f437b39c730 00007f437b46279b 00007f437b462821 00007f437b4262b0 00007f437b4117f9 00007f437b4112a2 00007f4374af7684
0925 03:04:31.897768 (+ 277us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:04:31.898113 (+ 345us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:31.898143 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:04:31.898575 (+ 432us) log.cc:842] Serialized 26 byte log entry
0925 03:04:31.927621 (+ 29046us) raft_consensus.cc:1631] Marking committed up to 2
0925 03:04:31.927666 (+ 45us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:31.927729 (+ 63us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:04:32.743458 (+815729us) raft_consensus.cc:1671] finished
0925 03:04:32.743480 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:32.772309 (+ 28829us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1565,"raft.run_cpu_time_us":161,"raft.run_wall_time_us":5055,"spinlock_wait_cycles":39063936,"thread_start_us":28699,"threads_started":1,"wal-append.queue_time_us":245384}
I20260925 03:04:32.887362 25805 raft_consensus.cc:504] T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:32.926839 25964 raft_consensus.cc:482] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:32.961719 25793 consensus_queue.cc:260] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:32.962468 26153 raft_consensus.cc:482] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:32.963431 26131 raft_consensus.cc:3037] T f22ddd7d1c444dc58f45a1fb0b3e02bc P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:32.964080 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c23337a7cb74184a0e57b7095cb6a11" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:33.003476 24468 raft_consensus.cc:2454] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 4.
I20260925 03:04:32.964496 24466 raft_consensus.cc:2454] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 5.
I20260925 03:04:32.947357 26086 raft_consensus.cc:482] T 442d4a29b1d743599f4b6b4871e26668 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:32.936954 26125 raft_consensus.cc:482] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:32.908852 25984 raft_consensus.cc:504] T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:32.976665 24437 leader_election.cc:302] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:04:32.925209 26094 consensus_queue.cc:1225] T a4abe0ea757e4886be4945c5001e529a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Peer eeab93abfdb141a895002ec9f312296c log is divergent from this leader: its last log entry 2.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260925 03:04:32.939312 26105 raft_consensus.cc:482] T 74443a0a1dc34d88a2f42e3b722db405 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:32.958302 26126 raft_consensus.cc:729] T 37a157996e944db2906110d87ecb0c54 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:04:32.993485 25964 raft_consensus.cc:504] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:33.054390 26152 consensus_queue.cc:1035] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.134s
I20260925 03:04:33.072530 25783 raft_consensus.cc:2741] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader election decision vote started in defunct term 4: won
I20260925 03:04:33.016824 26086 raft_consensus.cc:504] T 442d4a29b1d743599f4b6b4871e26668 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:33.007678 26107 leader_election.cc:288] T c88cbd39e01f44628a094af516be2833 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:32.916509 26147 consensus_queue.cc:1046] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.082s
I20260925 03:04:33.021217 25793 raft_consensus.cc:3037] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:32.984949 26136 raft_consensus.cc:504] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:32.979703 25982 raft_consensus.cc:504] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:33.053795 26151 raft_consensus.cc:482] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:32.914609 26123 raft_consensus.cc:504] T 420163a6cdce412ebc21c25fe3835638 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:33.048806 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7691c6b475428c82d6181d693d26f3" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
W20260925 03:04:33.073700 24585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.4:43979, user_credentials={real_user=slave}} blocked reactor thread for 71756.8us
I20260925 03:04:32.986517 26124 raft_consensus.cc:482] T edcf6e6a422d4d198d4fbb0f7d3d845d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:33.016516 26125 raft_consensus.cc:3037] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:33.043072 26121 raft_consensus.cc:3037] T 43366a0a2f9d4344bf4e37ffd290ef8a P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:33.113117 24546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 664) took 1575 ms. Trace:
I20260925 03:04:33.123492 26105 raft_consensus.cc:504] T 74443a0a1dc34d88a2f42e3b722db405 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:33.112138 26094 consensus_queue.cc:1046] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.012s
I20260925 03:04:32.971231 25197 raft_consensus.cc:2454] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 3.
I20260925 03:04:33.079579 26066 raft_consensus.cc:482] T ac9357d144aa49789faaf31fe5e6bf89 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:33.050457 26104 consensus_queue.cc:237] T 8ac466793d094325ace224708b2158ed P 1bc1f2e62ab44af995165bd5116a9d43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
W20260925 03:04:33.149251 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 182526us
W20260925 03:04:33.113755 25162 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 92626.3us
I20260925 03:04:33.002725 25987 leader_election.cc:288] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:32.954447 26071 raft_consensus.cc:504] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:33.158875 26147 consensus_queue.cc:1225] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Peer eeab93abfdb141a895002ec9f312296c log is divergent from this leader: its last log entry 4.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260925 03:04:33.159241 26134 consensus_queue.cc:1035] T e92f377a79354a6ebb4eec67303c13e3 P 1bc1f2e62ab44af995165bd5116a9d43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.019s
I20260925 03:04:33.137663 26163 raft_consensus.cc:482] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:33.023289 26153 raft_consensus.cc:504] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:33.150447 26140 raft_consensus.cc:482] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:33.150177 25783 raft_consensus.cc:482] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260925 03:04:33.259438 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 272142us
I20260925 03:04:33.373880 24546 rpcz_store.cc:276] 0925 03:04:31.537294 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:31.537451 (+ 157us) service_pool.cc:227] Handling call
0925 03:04:31.963633 (+426182us) spinlock_profiling.cc:243] Waited 1.14 ms on lock 0x7b5800003c98. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f43733ecf11 00007f4373570851 00007f4373570702 00007f437c460b09 00007f437b39c730 00007f437b46279b 00007f437b462821 00007f437b4262b0 00007f437b4117f9 00007f437b4112a2 00007f4374af7684
0925 03:04:31.963869 (+ 236us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:04:32.299487 (+335618us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:32.299520 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:04:32.324003 (+ 24483us) log.cc:842] Serialized 26 byte log entry
0925 03:04:32.629503 (+305500us) raft_consensus.cc:1631] Marking committed up to 2
0925 03:04:32.629540 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:32.629854 (+ 314us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:04:33.086617 (+456763us) raft_consensus.cc:1671] finished
0925 03:04:33.086640 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:33.113039 (+ 26399us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":23953,"raft.queue_time_us":108823,"raft.run_cpu_time_us":172,"raft.run_wall_time_us":8384,"spinlock_wait_cycles":2613504,"thread_start_us":356295,"threads_started":2,"wal-append.queue_time_us":340468}
W20260925 03:04:33.158296 24465 consensus_meta.cc:287] T 7993f9c3e2a44f408bbf5d1e395e2306 P 869b0a8575b54d08ae2817fad5dd42d5: Time spent flushing consensus metadata: real 0.858s user 0.013s sys 0.000s
I20260925 03:04:33.378640 25979 raft_consensus.cc:482] T 37a157996e944db2906110d87ecb0c54 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260925 03:04:33.379454 26133 log.cc:924] Time spent T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5: Append to log took a long time: real 0.362s user 0.002s sys 0.000s
I20260925 03:04:33.380553 26086 leader_election.cc:288] T 442d4a29b1d743599f4b6b4871e26668 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:33.381659 26136 leader_election.cc:288] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:33.375869 26119 leader_election.cc:288] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
W20260925 03:04:33.382505 26080 log.cc:924] Time spent T 48ca6ccd98194cf2af7b5278d15bf4ea P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 1.791s user 0.007s sys 0.005s
I20260925 03:04:33.300014 26152 consensus_queue.cc:1035] T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.091s
W20260925 03:04:33.383311 26154 log.cc:924] Time spent T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.398s user 0.002s sys 0.000s
I20260925 03:04:33.385926 24548 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33068 (request call id 87) took 1492 ms. Trace:
W20260925 03:04:33.386844 26068 consensus_meta.cc:287] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c: Time spent flushing consensus metadata: real 0.658s user 0.005s sys 0.006s
W20260925 03:04:33.363334 26161 log.cc:924] Time spent T 3d89396576884f7c8901dba54f783947 P 869b0a8575b54d08ae2817fad5dd42d5: Append to log took a long time: real 0.203s user 0.002s sys 0.000s
I20260925 03:04:33.379539 25982 leader_election.cc:288] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:33.242877 25897 leader_election.cc:288] T 3d89396576884f7c8901dba54f783947 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
W20260925 03:04:33.388106 26132 log.cc:924] Time spent T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.314s user 0.000s sys 0.001s
W20260925 03:04:33.389551 26158 log.cc:924] Time spent T 1414dcfe53f64a78ae413531eb8ec117 P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.233s user 0.001s sys 0.000s
W20260925 03:04:33.388998 25197 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48424 (request call id 173) took 1167 ms (client timeout 1505 ms). Trace:
W20260925 03:04:33.160055 26144 consensus_queue.cc:1159] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "eeab93abfdb141a895002ec9f312296c" responder_term: 4 status { last_received { term: 4 index: 3 } last_committed_idx: 2 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 3. Current term is 4. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
W20260925 03:04:33.393658 26142 log.cc:924] Time spent T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.249s user 0.002s sys 0.000s
I20260925 03:04:33.394747 26159 raft_consensus.cc:482] T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
W20260925 03:04:33.382302 25973 consensus_meta.cc:287] T b1e85248d3b04b979dd080caeeb02fdb P eeab93abfdb141a895002ec9f312296c: Time spent flushing consensus metadata: real 0.708s user 0.008s sys 0.004s
W20260925 03:04:33.309898 24541 consensus_meta.cc:287] T 7993f9c3e2a44f408bbf5d1e395e2306 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent flushing consensus metadata: real 0.565s user 0.008s sys 0.006s
W20260925 03:04:33.318526 26118 consensus_meta.cc:287] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent flushing consensus metadata: real 1.071s user 0.018s sys 0.000s
W20260925 03:04:33.261318 26130 log.cc:924] Time spent T 0b5df303cd6a4d59aca5c18bbbff24ed P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.267s user 0.000s sys 0.001s
W20260925 03:04:33.195960 25953 consensus_meta.cc:287] T 21c43af5e6844fdaacc8c625fa01fd8d P 869b0a8575b54d08ae2817fad5dd42d5: Time spent flushing consensus metadata: real 1.001s user 0.009s sys 0.008s
I20260925 03:04:33.467139 26160 raft_consensus.cc:482] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:33.264106 26126 consensus_queue.cc:260] T 37a157996e944db2906110d87ecb0c54 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:33.425933 26119 raft_consensus.cc:482] T 0be4e0aa984f4f2b81a58e382e434ad5 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:33.426645 26101 raft_consensus.cc:482] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:33.426995 25890 raft_consensus.cc:482] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:33.491539 26136 raft_consensus.cc:482] T 0b5df303cd6a4d59aca5c18bbbff24ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:33.276998 25960 raft_consensus.cc:482] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:33.386746 26162 raft_consensus.cc:482] T 29637f5a4a504ce6ba29033ccd0cd716 P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:33.450392 24544 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 668) took 1606 ms. Trace:
I20260925 03:04:33.377658 24539 raft_consensus.cc:3037] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:33.460408 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c88cbd39e01f44628a094af516be2833" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:33.495616 24541 raft_consensus.cc:2454] T 7993f9c3e2a44f408bbf5d1e395e2306 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 3.
W20260925 03:04:33.467185 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 66413.4us
I20260925 03:04:33.450392 24548 rpcz_store.cc:276] 0925 03:04:31.893826 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:32.101807 (+207981us) service_pool.cc:227] Handling call
0925 03:04:32.135288 (+ 33481us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:04:32.157244 (+ 21956us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:32.157281 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:04:32.178525 (+ 21244us) log.cc:842] Serialized 26 byte log entry
0925 03:04:32.958842 (+780317us) raft_consensus.cc:1631] Marking committed up to 6
0925 03:04:32.989387 (+ 30545us) log.cc:842] Serialized 14 byte log entry
0925 03:04:32.989609 (+ 222us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:32.989679 (+ 70us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:04:33.385687 (+396008us) raft_consensus.cc:1671] finished
0925 03:04:33.385713 (+ 26us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:33.385857 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7849,"prepare.queue_time_us":191588,"prepare.run_cpu_time_us":10268,"prepare.run_wall_time_us":34346,"raft.queue_time_us":153321,"raft.run_cpu_time_us":525,"raft.run_wall_time_us":25885,"thread_start_us":705179,"threads_started":1,"wal-append.queue_time_us":977582}
W20260925 03:04:33.491385 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 84012.3us
I20260925 03:04:33.460842 26070 raft_consensus.cc:482] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:33.423002 25783 raft_consensus.cc:504] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:33.424021 25979 raft_consensus.cc:504] T 37a157996e944db2906110d87ecb0c54 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:33.536029 26070 raft_consensus.cc:504] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:33.538461 25783 leader_election.cc:288] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:33.344069 25984 leader_election.cc:288] T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:33.465783 26147 raft_consensus.cc:482] T b030d84c670b48d5bff337ccc0c77626 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:33.538903 25979 leader_election.cc:288] T 37a157996e944db2906110d87ecb0c54 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:33.399655 26066 raft_consensus.cc:504] T ac9357d144aa49789faaf31fe5e6bf89 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:33.468020 25897 raft_consensus.cc:482] T 06f4a7eaf6534e4986c9fd35911038e4 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:33.545163 25897 raft_consensus.cc:504] T 06f4a7eaf6534e4986c9fd35911038e4 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:33.377112 26124 raft_consensus.cc:504] T edcf6e6a422d4d198d4fbb0f7d3d845d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:33.551937 26124 leader_election.cc:288] T edcf6e6a422d4d198d4fbb0f7d3d845d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:33.309499 25964 leader_election.cc:288] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:33.512023 24472 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 725) took 1199 ms. Trace:
I20260925 03:04:33.554229 24472 rpcz_store.cc:276] 0925 03:04:32.312396 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:32.312537 (+ 141us) service_pool.cc:227] Handling call
0925 03:04:32.381415 (+ 68878us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:04:32.382822 (+ 1407us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:32.382854 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:04:32.442925 (+ 60071us) log.cc:842] Serialized 26 byte log entry
0925 03:04:32.645630 (+202705us) raft_consensus.cc:1631] Marking committed up to 3
0925 03:04:32.645665 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:32.645729 (+ 64us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:04:33.511728 (+865999us) raft_consensus.cc:1671] finished
0925 03:04:33.511750 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:33.511943 (+ 193us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1037,"raft.queue_time_us":1178,"raft.run_cpu_time_us":148,"raft.run_wall_time_us":56664,"thread_start_us":182504,"threads_started":1,"wal-append.queue_time_us":509528}
W20260925 03:04:33.508601 25983 consensus_meta.cc:287] T c88cbd39e01f44628a094af516be2833 P eeab93abfdb141a895002ec9f312296c: Time spent flushing consensus metadata: real 0.776s user 0.013s sys 0.001s
I20260925 03:04:33.556293 25983 raft_consensus.cc:504] T c88cbd39e01f44628a094af516be2833 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:04:33.466240 25197 rpcz_store.cc:269] 0925 03:04:32.221712 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:32.235843 (+ 14131us) service_pool.cc:227] Handling call
0925 03:04:33.388918 (+1153075us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:33.421576 24465 raft_consensus.cc:2454] T 7993f9c3e2a44f408bbf5d1e395e2306 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 3.
I20260925 03:04:33.561712 24465 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 722) took 1595 ms. Trace:
I20260925 03:04:33.562420 24465 rpcz_store.cc:276] 0925 03:04:31.965969 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:31.972257 (+ 6288us) service_pool.cc:227] Handling call
0925 03:04:33.561552 (+1589295us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:33.530143 24546 raft_consensus.cc:2454] T c88cbd39e01f44628a094af516be2833 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 4.
I20260925 03:04:33.565316 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c23337a7cb74184a0e57b7095cb6a11" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:33.565990 24546 raft_consensus.cc:2454] T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 4.
I20260925 03:04:33.529848 24544 rpcz_store.cc:276] 0925 03:04:31.843979 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:31.844098 (+ 119us) service_pool.cc:227] Handling call
0925 03:04:31.844444 (+ 346us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:04:31.881345 (+ 36901us) raft_consensus.cc:1520] Early marking committed up to index 3
0925 03:04:32.659477 (+778132us) spinlock_profiling.cc:243] Waited 13.5 ms on lock 0x7b18000ffbe0. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f437c4478b1 00007f437c454661 00007f437358a345 00007f437c3c1a39 00007f43735a78bb 00007f43735a5cbf 00007f43735a5155 00007f4379ea5167 00007f43764da807 00007f43764e9f0f 00007f43764e9e88
0925 03:04:32.946908 (+287431us) log.cc:842] Serialized 14 byte log entry
0925 03:04:33.195770 (+248862us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:04:33.286572 (+ 90802us) log.cc:842] Serialized 26 byte log entry
0925 03:04:33.286727 (+ 155us) raft_consensus.cc:1631] Marking committed up to 3
0925 03:04:33.286763 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:33.286831 (+ 68us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:04:33.428064 (+141233us) raft_consensus.cc:1671] finished
0925 03:04:33.450143 (+ 22079us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:33.450326 (+ 183us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":52271,"prepare.queue_time_us":563100,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":39,"raft.queue_time_us":285367,"raft.run_cpu_time_us":389,"raft.run_wall_time_us":21766,"spinlock_wait_cycles":30941952,"thread_start_us":401868,"threads_started":2,"wal-append.queue_time_us":465672}
W20260925 03:04:33.575150 24511 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.4:43979, user_credentials={real_user=slave}} blocked reactor thread for 77999.5us
I20260925 03:04:33.560201 25983 leader_election.cc:288] T c88cbd39e01f44628a094af516be2833 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:33.582562 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e5819c15eac4544a57e5ee6c9d4f117" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:33.585420 24548 raft_consensus.cc:2454] T 8e5819c15eac4544a57e5ee6c9d4f117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 1.
I20260925 03:04:33.460103 26068 raft_consensus.cc:504] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:33.525269 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c81a1b3b616492ab6c964f45af36077" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:33.495239 25973 raft_consensus.cc:504] T b1e85248d3b04b979dd080caeeb02fdb P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:33.398885 26163 raft_consensus.cc:504] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:04:33.577996 26132 log.cc:924] Time spent T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.056s user 0.000s sys 0.001s
I20260925 03:04:33.401678 26140 raft_consensus.cc:504] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:33.242843 26151 raft_consensus.cc:504] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:33.596225 24588 leader_election.cc:302] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:33.571888 25957 raft_consensus.cc:2741] T 7993f9c3e2a44f408bbf5d1e395e2306 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20260925 03:04:33.570173 26170 raft_consensus.cc:482] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:33.530997 26136 raft_consensus.cc:504] T 0b5df303cd6a4d59aca5c18bbbff24ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:33.522719 26119 raft_consensus.cc:504] T 0be4e0aa984f4f2b81a58e382e434ad5 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:33.532815 25960 raft_consensus.cc:504] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
W20260925 03:04:33.595561 26130 log.cc:924] Time spent T 0b5df303cd6a4d59aca5c18bbbff24ed P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.062s user 0.000s sys 0.000s
I20260925 03:04:33.591898 25897 leader_election.cc:288] T 06f4a7eaf6534e4986c9fd35911038e4 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
W20260925 03:04:33.622778 24585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.4:43979, user_credentials={real_user=slave}} blocked reactor thread for 63171.1us
I20260925 03:04:33.595343 26126 raft_consensus.cc:3037] T 37a157996e944db2906110d87ecb0c54 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:33.605803 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b8e0d260a884ecdb20f79de46c36b19" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:33.586735 26168 raft_consensus.cc:482] T a7aa47c48404465b94a36fd819fe3c7a P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:33.593634 24540 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 657) took 2265 ms. Trace:
I20260925 03:04:33.528954 25890 raft_consensus.cc:504] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:33.521613 26160 raft_consensus.cc:504] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:33.539494 26147 raft_consensus.cc:504] T b030d84c670b48d5bff337ccc0c77626 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:33.600364 26066 leader_election.cc:288] T ac9357d144aa49789faaf31fe5e6bf89 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:33.632404 24548 raft_consensus.cc:2454] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 2.
I20260925 03:04:33.634972 24543 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33068 (request call id 86) took 1741 ms. Trace:
I20260925 03:04:33.619880 26070 leader_election.cc:288] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:33.497890 26159 raft_consensus.cc:504] T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:33.599455 26171 raft_consensus.cc:3032] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 LEADER]: Stepping down as leader of term 3
I20260925 03:04:33.626652 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c88cbd39e01f44628a094af516be2833" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:33.624747 26105 leader_election.cc:288] T 74443a0a1dc34d88a2f42e3b722db405 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:33.652063 24543 rpcz_store.cc:276] 0925 03:04:31.893552 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:32.098970 (+205418us) service_pool.cc:227] Handling call
0925 03:04:32.149596 (+ 50626us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:04:32.198731 (+ 49135us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:32.198763 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:04:32.218343 (+ 19580us) log.cc:842] Serialized 26 byte log entry
0925 03:04:32.620986 (+402643us) raft_consensus.cc:1631] Marking committed up to 5
0925 03:04:32.631975 (+ 10989us) log.cc:842] Serialized 14 byte log entry
0925 03:04:32.719219 (+ 87244us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:32.719522 (+ 303us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:04:33.634750 (+915228us) raft_consensus.cc:1671] finished
0925 03:04:33.634773 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:33.634910 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":29711,"prepare.queue_time_us":27190,"prepare.run_cpu_time_us":73,"prepare.run_wall_time_us":6566,"raft.queue_time_us":162310,"raft.run_cpu_time_us":171,"raft.run_wall_time_us":171,"thread_start_us":430348,"threads_started":2,"wal-append.queue_time_us":738785}
I20260925 03:04:33.631834 26123 leader_election.cc:288] T 420163a6cdce412ebc21c25fe3835638 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:33.661374 24545 raft_consensus.cc:1261] T 3d89396576884f7c8901dba54f783947 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260925 03:04:33.667901 24545 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 666) took 2104 ms. Trace:
I20260925 03:04:33.668161 24545 rpcz_store.cc:276] 0925 03:04:31.563331 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:31.565869 (+ 2538us) service_pool.cc:227] Handling call
0925 03:04:33.660833 (+2094964us) spinlock_profiling.cc:243] Waited 1.11 ms on lock 0x7b5800003c98. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f43733ecf11 00007f4373570851 00007f4373570702 00007f437c460b09 00007f437b39c730 00007f437b46279b 00007f437b462821 00007f437b4262b0 00007f437b4117f9 00007f437b4112a2 00007f4374af7684
0925 03:04:33.661098 (+ 265us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:33.667698 (+ 6600us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:33.667858 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2563840}
I20260925 03:04:33.669032 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60d45b7d467d4e1ab92367f6d7d52b15" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:33.635979 24616 raft_consensus.cc:1226] T b1e85248d3b04b979dd080caeeb02fdb P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Rejecting Update request from peer 1bc1f2e62ab44af995165bd5116a9d43 for earlier term 2. Current term is 3. Ops: []
I20260925 03:04:33.633855 25897 raft_consensus.cc:482] T ceab6b0fccd54e909a64f66fe81a9a1f P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:33.646001 26167 raft_consensus.cc:482] T f22ddd7d1c444dc58f45a1fb0b3e02bc P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:33.651116 26071 leader_election.cc:288] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:33.665038 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "550118bc1a0641ce9803bdbf43f2069f" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:33.686357 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37a157996e944db2906110d87ecb0c54" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:33.644579 25190 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48424 (request call id 174) took 1422 ms. Trace:
I20260925 03:04:33.640285 26160 leader_election.cc:288] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:33.653322 26118 raft_consensus.cc:482] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1bc1f2e62ab44af995165bd5116a9d43)
I20260925 03:04:33.656147 26171 raft_consensus.cc:729] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:04:33.694903 24548 raft_consensus.cc:2454] T 550118bc1a0641ce9803bdbf43f2069f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 2.
I20260925 03:04:33.655323 25960 leader_election.cc:288] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:33.682942 26070 raft_consensus.cc:482] T 5278a60f97da4e10aecd50a8d9a9a43c P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260925 03:04:33.673660 24587 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.521s user 0.001s sys 0.010s
I20260925 03:04:33.698696 26177 raft_consensus.cc:482] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:33.689350 26105 raft_consensus.cc:2781] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260925 03:04:33.716835 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc2df40db08649c9a75e90000b878559" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:33.667196 26083 raft_consensus.cc:482] T c85f5d1d3f004cf5817bb594fd8e6469 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:33.525071 26101 raft_consensus.cc:504] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
W20260925 03:04:33.715198 26169 log.cc:924] Time spent T 367277daf8d04778b3fa4ed94d71580d P 1bc1f2e62ab44af995165bd5116a9d43: Append to log took a long time: real 0.209s user 0.002s sys 0.000s
I20260925 03:04:33.574407 26162 raft_consensus.cc:504] T 29637f5a4a504ce6ba29033ccd0cd716 P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:33.705778 25973 leader_election.cc:288] T b1e85248d3b04b979dd080caeeb02fdb P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 3 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:33.666893 24541 raft_consensus.cc:2454] T c88cbd39e01f44628a094af516be2833 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 4.
I20260925 03:04:33.628227 26170 raft_consensus.cc:504] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:33.689831 25190 rpcz_store.cc:276] 0925 03:04:32.222029 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:32.223948 (+ 1919us) service_pool.cc:227] Handling call
0925 03:04:32.224325 (+ 377us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:32.455109 (+230784us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:33.644072 (+1188963us) spinlock_profiling.cc:243] Waited 76 ms on lock 0x7b54005c08d4. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d0f51 00007f43764cf99e 00007f437b4112e9 00007f4374af7684 00007f4374af7612 00007f4374af7541 00007f4374af74bc 00007f4374af6772 00007f43748fa94c 00007f43748f9d86
0925 03:04:33.644457 (+ 385us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":174725888}
I20260925 03:04:33.717492 24548 raft_consensus.cc:3037] T cc2df40db08649c9a75e90000b878559 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:33.694865 25953 raft_consensus.cc:482] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:33.721227 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e725b82985746b39c0cc8f671c95353" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
W20260925 03:04:33.708897 24587 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.564s user 0.001s sys 0.010s
I20260925 03:04:33.691116 25897 raft_consensus.cc:504] T ceab6b0fccd54e909a64f66fe81a9a1f P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:04:33.719161 25191 consensus_meta.cc:287] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43: Time spent flushing consensus metadata: real 1.119s user 0.014s sys 0.001s
I20260925 03:04:33.713158 25960 raft_consensus.cc:482] T b4ebd2b043024a548005c8b6b00b0449 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1bc1f2e62ab44af995165bd5116a9d43)
I20260925 03:04:33.718750 26105 raft_consensus.cc:482] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:33.738524 26105 raft_consensus.cc:3037] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:33.741675 26170 leader_election.cc:288] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:33.649195 26168 raft_consensus.cc:504] T a7aa47c48404465b94a36fd819fe3c7a P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:33.746687 26168 leader_election.cc:288] T a7aa47c48404465b94a36fd819fe3c7a P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:33.721805 26140 leader_election.cc:288] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:33.712518 26070 raft_consensus.cc:504] T 5278a60f97da4e10aecd50a8d9a9a43c P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:33.754308 25199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1e85248d3b04b979dd080caeeb02fdb" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43"
I20260925 03:04:33.753943 26119 leader_election.cc:288] T 0be4e0aa984f4f2b81a58e382e434ad5 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:33.728745 26068 leader_election.cc:288] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 3 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:33.754472 25960 raft_consensus.cc:504] T b4ebd2b043024a548005c8b6b00b0449 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:33.718295 26118 raft_consensus.cc:504] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:33.698256 26174 consensus_queue.cc:1035] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.157s
I20260925 03:04:33.730643 26178 raft_consensus.cc:482] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1bc1f2e62ab44af995165bd5116a9d43)
I20260925 03:04:33.756286 25191 raft_consensus.cc:2454] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 5.
I20260925 03:04:33.766479 26153 leader_election.cc:288] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:33.736147 25953 raft_consensus.cc:504] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:33.769675 26171 consensus_queue.cc:260] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:33.725525 26083 raft_consensus.cc:504] T c85f5d1d3f004cf5817bb594fd8e6469 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:33.822301 25960 leader_election.cc:288] T b4ebd2b043024a548005c8b6b00b0449 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:33.786079 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:33.635468 24540 rpcz_store.cc:276] 0925 03:04:31.314080 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:31.318703 (+ 4623us) service_pool.cc:227] Handling call
0925 03:04:31.370187 (+ 51484us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:04:31.370685 (+ 498us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:31.370716 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:04:31.371167 (+ 451us) log.cc:842] Serialized 26 byte log entry
0925 03:04:31.488078 (+116911us) raft_consensus.cc:1631] Marking committed up to 1
0925 03:04:31.491469 (+ 3391us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:31.497697 (+ 6228us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:04:31.646704 (+149007us) spinlock_profiling.cc:243] Waited 3.68 ms on lock 0x7b48000c1ed8. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f437b3777f1 00007f437b36e8e7 00007f437b36e77a 00007f437b46295f 00007f437b462627 00007f437b476b2c 00007f437b476ad8 00007f437b476a58 00007f437b476a13 00007f437b475cbc 00007f437a3f7be6
0925 03:04:33.381463 (+1734759us) spinlock_profiling.cc:243] Waited 2.09 ms on lock 0x7b44000232cc. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f43733ecf11 00007f43734baa95 00007f43734bb4b2 00007f4376163511 00007f437615ee1a 00007f4376152bee 00007f4376151f36 00007f4376151b58 00007f437615b551 00007f437615b509 00007f437615b499
0925 03:04:33.533321 (+151858us) raft_consensus.cc:1671] finished
0925 03:04:33.570369 (+ 37048us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:33.579724 (+ 9355us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":117,"raft.run_cpu_time_us":1844,"raft.run_wall_time_us":276207,"spinlock_wait_cycles":13268096,"thread_start_us":108832,"threads_started":1,"wal-append.queue_time_us":192223}
I20260925 03:04:33.829843 25193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e92f377a79354a6ebb4eec67303c13e3" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:33.790354 26136 leader_election.cc:288] T 0b5df303cd6a4d59aca5c18bbbff24ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:33.854077 25960 raft_consensus.cc:482] T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:33.854621 26189 raft_consensus.cc:482] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:33.849637 26163 leader_election.cc:288] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:33.866329 26181 raft_consensus.cc:482] T 550118bc1a0641ce9803bdbf43f2069f P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:33.865713 25960 raft_consensus.cc:504] T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:04:33.818696 25191 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49794 (request call id 152) took 1483 ms (client timeout 1670 ms). Trace:
I20260925 03:04:33.871138 24615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c23337a7cb74184a0e57b7095cb6a11" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:33.725793 26177 raft_consensus.cc:504] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:04:33.874745 25788 raft_consensus.cc:659] T 367277daf8d04778b3fa4ed94d71580d P 1bc1f2e62ab44af995165bd5116a9d43: failed to trigger leader election: Illegal state: only voting members can start elections: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } }
I20260925 03:04:33.875464 24622 raft_consensus.cc:1261] T cc2df40db08649c9a75e90000b878559 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260925 03:04:33.875276 24618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b030d84c670b48d5bff337ccc0c77626" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:33.877348 24618 raft_consensus.cc:3037] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:33.816273 25194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ac466793d094325ace224708b2158ed" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:33.857923 24541 raft_consensus.cc:3037] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:33.876287 25960 leader_election.cc:288] T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:33.885108 24548 raft_consensus.cc:2454] T cc2df40db08649c9a75e90000b878559 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 3.
I20260925 03:04:33.885299 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b8e0d260a884ecdb20f79de46c36b19" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
W20260925 03:04:33.886593 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 290443us
I20260925 03:04:33.896194 24588 leader_election.cc:302] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:33.870535 26171 raft_consensus.cc:3037] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:33.899561 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442d4a29b1d743599f4b6b4871e26668" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:33.922439 25199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:33.923146 25199 raft_consensus.cc:2454] T 29637f5a4a504ce6ba29033ccd0cd716 P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 3.
I20260925 03:04:33.921666 24548 raft_consensus.cc:2454] T 442d4a29b1d743599f4b6b4871e26668 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 4.
I20260925 03:04:33.702984 26167 raft_consensus.cc:504] T f22ddd7d1c444dc58f45a1fb0b3e02bc P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:33.866391 25897 leader_election.cc:288] T ceab6b0fccd54e909a64f66fe81a9a1f P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:33.905821 24540 raft_consensus.cc:3037] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:33.895107 26119 raft_consensus.cc:482] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:33.917860 26070 leader_election.cc:288] T 5278a60f97da4e10aecd50a8d9a9a43c P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:33.886235 26163 raft_consensus.cc:482] T b4ebd2b043024a548005c8b6b00b0449 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1bc1f2e62ab44af995165bd5116a9d43)
I20260925 03:04:33.886475 26136 raft_consensus.cc:482] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260925 03:04:33.870939 25191 rpcz_store.cc:269] 0925 03:04:32.327488 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:32.361177 (+ 33689us) service_pool.cc:227] Handling call
0925 03:04:33.810656 (+1449479us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:33.974347 24588 leader_election.cc:302] T 7993f9c3e2a44f408bbf5d1e395e2306 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:33.866899 26181 raft_consensus.cc:504] T 550118bc1a0641ce9803bdbf43f2069f P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:33.875517 26189 raft_consensus.cc:504] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:33.927899 26123 raft_consensus.cc:2781] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Leader election won for term 5
I20260925 03:04:33.984187 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442d4a29b1d743599f4b6b4871e26668" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:33.931255 24614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5f9b503b0ab43999d582015a367441e" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:33.991912 24618 raft_consensus.cc:2454] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 5.
I20260925 03:04:33.979043 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5332a5d329493991dff994e70ea92c" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
W20260925 03:04:34.080842 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 106601us
I20260925 03:04:34.086196 26101 leader_election.cc:288] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:33.974619 26068 raft_consensus.cc:482] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:34.086621 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5278a60f97da4e10aecd50a8d9a9a43c" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:33.844653 26178 raft_consensus.cc:504] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
W20260925 03:04:34.041127 26187 log.cc:924] Time spent T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.170s user 0.001s sys 0.000s
W20260925 03:04:33.932082 25793 consensus_meta.cc:287] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent flushing consensus metadata: real 0.776s user 0.009s sys 0.004s
I20260925 03:04:33.931963 25196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "293ef69caf514ac98510a68495f54eda" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43"
I20260925 03:04:33.930539 26186 raft_consensus.cc:482] T b627ecb4a5384ef392aa695e06599c39 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:33.984633 26136 raft_consensus.cc:504] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:04:34.187664 24586 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.592s user 0.030s sys 0.029s
W20260925 03:04:34.294246 24547 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36786 (request call id 662) took 2808 ms (client timeout 1630 ms). Trace:
W20260925 03:04:34.302142 24586 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.706s user 0.034s sys 0.033s
W20260925 03:04:33.983623 26183 log.cc:924] Time spent T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.250s user 0.001s sys 0.001s
I20260925 03:04:34.295928 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5332a5d329493991dff994e70ea92c" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:34.054039 25960 raft_consensus.cc:482] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:33.980441 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c81a1b3b616492ab6c964f45af36077" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:34.040954 26118 leader_election.cc:288] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
W20260925 03:04:34.302342 24547 rpcz_store.cc:269] 0925 03:04:31.485441 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:31.485587 (+ 146us) service_pool.cc:227] Handling call
0925 03:04:34.293974 (+2808387us) spinlock_profiling.cc:243] Waited 135 ms on lock 0x7b54002814d4. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d42ae 00007f437b411c27 00007f4374af9c04 00007f4374af9b92 00007f4374af9ac1 00007f4374af9a3c 00007f4374af8cf2 00007f43748fa94c 00007f43748f9d86 00007f43748f972f
0925 03:04:34.294118 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":309931392}
I20260925 03:04:34.302165 24618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e54ca1757ee4e42956a9cebbeca4f62" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:34.156361 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be4e0aa984f4f2b81a58e382e434ad5" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:34.038666 26162 leader_election.cc:288] T 29637f5a4a504ce6ba29033ccd0cd716 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:33.993129 26159 leader_election.cc:288] T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:34.331597 24621 raft_consensus.cc:2454] T 5278a60f97da4e10aecd50a8d9a9a43c P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 2.
I20260925 03:04:34.306233 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "293ef69caf514ac98510a68495f54eda" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:34.168874 24622 raft_consensus.cc:1261] T 550118bc1a0641ce9803bdbf43f2069f P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260925 03:04:34.317400 24542 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36786 (request call id 674) took 1624 ms. Trace:
I20260925 03:04:34.345037 24542 rpcz_store.cc:276] 0925 03:04:32.680962 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:32.681112 (+ 150us) service_pool.cc:227] Handling call
0925 03:04:34.295715 (+1614603us) spinlock_profiling.cc:243] Waited 2.9 ms on lock 0x7b5800003c98. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f43733ecf11 00007f4373570851 00007f4373570702 00007f437c460b09 00007f437b39c730 00007f437b46279b 00007f437b462821 00007f437b4121b0 00007f437b411a94 00007f4374af9c04 00007f4374af9b92
0925 03:04:34.305476 (+ 9761us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6659840}
I20260925 03:04:33.966432 26151 leader_election.cc:288] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:33.931095 26172 consensus_queue.cc:1046] T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.303s
I20260925 03:04:34.026733 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d89396576884f7c8901dba54f783947" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
W20260925 03:04:34.271638 26131 consensus_meta.cc:287] T f22ddd7d1c444dc58f45a1fb0b3e02bc P 869b0a8575b54d08ae2817fad5dd42d5: Time spent flushing consensus metadata: real 1.220s user 0.011s sys 0.006s
I20260925 03:04:34.166381 24614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06f4a7eaf6534e4986c9fd35911038e4" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
W20260925 03:04:34.100112 24539 consensus_meta.cc:287] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent flushing consensus metadata: real 0.514s user 0.010s sys 0.003s
I20260925 03:04:34.359144 24548 raft_consensus.cc:2454] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 4.
I20260925 03:04:34.359453 24586 leader_election.cc:302] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
W20260925 03:04:34.305236 26104 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.575s user 0.032s sys 0.037s
I20260925 03:04:34.086675 25194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71f461a6e2ff434993b78a2d077f0abd" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:34.352612 26078 raft_consensus.cc:3037] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:34.352932 26070 raft_consensus.cc:482] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:34.141971 26123 raft_consensus.cc:686] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [term 5 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:34.362008 26123 consensus_queue.cc:237] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:34.360867 24539 raft_consensus.cc:2454] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 5.
I20260925 03:04:34.366361 25890 leader_election.cc:288] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
W20260925 03:04:34.365744 24539 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36786 (request call id 685) took 1455 ms (client timeout 1507 ms). Trace:
I20260925 03:04:34.367443 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e92f377a79354a6ebb4eec67303c13e3" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
W20260925 03:04:34.367060 24539 rpcz_store.cc:269] 0925 03:04:32.910820 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:33.000246 (+ 89426us) service_pool.cc:227] Handling call
0925 03:04:34.365668 (+1365422us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:34.368672 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c23337a7cb74184a0e57b7095cb6a11" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:34.369285 24542 raft_consensus.cc:2454] T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 4.
W20260925 03:04:34.347380 26120 consensus_meta.cc:287] T 100c113a9fd84cc8adf2ec7b33ea0093 P eeab93abfdb141a895002ec9f312296c: Time spent flushing consensus metadata: real 1.398s user 0.009s sys 0.009s
I20260925 03:04:34.351596 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:34.212453 26167 leader_election.cc:288] T f22ddd7d1c444dc58f45a1fb0b3e02bc P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
W20260925 03:04:34.044231 24585 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.22.193.4:43979 timed out after 1.670s (SENT) [suppressed 29 similar messages]
I20260925 03:04:34.099741 25973 raft_consensus.cc:2781] T 7993f9c3e2a44f408bbf5d1e395e2306 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader election won for term 3
W20260925 03:04:34.350307 26071 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.560s user 0.004s sys 0.016s
I20260925 03:04:34.088335 26083 leader_election.cc:288] T c85f5d1d3f004cf5817bb594fd8e6469 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:34.087248 25953 leader_election.cc:288] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:34.275961 26193 raft_consensus.cc:482] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:34.361212 26070 raft_consensus.cc:504] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:34.300206 26186 raft_consensus.cc:504] T b627ecb4a5384ef392aa695e06599c39 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:34.003219 26163 raft_consensus.cc:504] T b4ebd2b043024a548005c8b6b00b0449 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:34.349509 24471 raft_consensus.cc:2379] T 293ef69caf514ac98510a68495f54eda P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 3: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
W20260925 03:04:34.297367 26192 log.cc:924] Time spent T a4abe0ea757e4886be4945c5001e529a P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.237s user 0.000s sys 0.001s
I20260925 03:04:34.039856 26174 raft_consensus.cc:482] T 21c43af5e6844fdaacc8c625fa01fd8d P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:34.368999 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b030d84c670b48d5bff337ccc0c77626" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:34.247368 26177 leader_election.cc:288] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:34.377341 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc2df40db08649c9a75e90000b878559" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:34.355309 24472 raft_consensus.cc:2354] T 0be4e0aa984f4f2b81a58e382e434ad5 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 2. Current term is 3.
I20260925 03:04:34.229382 26181 leader_election.cc:288] T 550118bc1a0641ce9803bdbf43f2069f P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:34.161607 24465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5f9b503b0ab43999d582015a367441e" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:34.173491 25191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:33.984876 26119 raft_consensus.cc:504] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:34.386992 26178 leader_election.cc:288] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:34.390403 25193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6b6ec010caf4689b8c90449764f5cf9" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
W20260925 03:04:34.387799 24588 connection.cc:232] RPC call timeout handler was delayed by -1.3596014559989271s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.163s for a response
I20260925 03:04:34.398216 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5f9b503b0ab43999d582015a367441e" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:34.413298 25973 raft_consensus.cc:686] T 7993f9c3e2a44f408bbf5d1e395e2306 P eeab93abfdb141a895002ec9f312296c [term 3 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:34.443948 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37a157996e944db2906110d87ecb0c54" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
W20260925 03:04:34.371368 26104 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.643s user 0.032s sys 0.037s
I20260925 03:04:34.321774 25196 raft_consensus.cc:2362] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 3. Re-sending same reply.
I20260925 03:04:34.372606 26131 raft_consensus.cc:2726] T f22ddd7d1c444dc58f45a1fb0b3e02bc P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260925 03:04:34.415599 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f68804eb5a244d87bb395d50683c6012" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:34.383142 26172 consensus_queue.cc:1225] T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Peer 1bc1f2e62ab44af995165bd5116a9d43 log is divergent from this leader: its last log entry 3.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260925 03:04:34.322180 25960 raft_consensus.cc:504] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:04:34.396777 26071 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.607s user 0.004s sys 0.016s
I20260925 03:04:34.426676 24548 raft_consensus.cc:2362] T cc2df40db08649c9a75e90000b878559 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 3. Re-sending same reply.
I20260925 03:04:34.420940 24469 raft_consensus.cc:1226] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 LEADER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 3. Current term is 4. Ops: []
I20260925 03:04:34.034103 26105 raft_consensus.cc:504] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:34.444340 24542 raft_consensus.cc:2454] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 4.
I20260925 03:04:34.385052 26120 raft_consensus.cc:504] T 100c113a9fd84cc8adf2ec7b33ea0093 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:34.427101 26174 raft_consensus.cc:504] T 21c43af5e6844fdaacc8c625fa01fd8d P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:34.453189 24547 raft_consensus.cc:2396] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
W20260925 03:04:34.429553 26113 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.607s user 0.005s sys 0.031s
I20260925 03:04:34.407513 26193 raft_consensus.cc:504] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:34.427703 26136 leader_election.cc:288] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:34.458814 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06f4a7eaf6534e4986c9fd35911038e4" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:34.506573 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d89396576884f7c8901dba54f783947" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:34.507231 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e8af2641e124f9589c5ccfdac785f63" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
W20260925 03:04:34.459767 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 100369us
W20260925 03:04:34.460656 24585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.4:43979, user_credentials={real_user=slave}} blocked reactor thread for 416437us
I20260925 03:04:34.438028 26167 raft_consensus.cc:482] T 8ac466793d094325ace224708b2158ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:34.437269 26177 raft_consensus.cc:482] T 1414dcfe53f64a78ae413531eb8ec117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:34.470019 26199 raft_consensus.cc:482] T 293ef69caf514ac98510a68495f54eda P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:34.481582 24615 raft_consensus.cc:2379] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 in current term 5: Already voted for candidate eeab93abfdb141a895002ec9f312296c in this term.
I20260925 03:04:34.482126 26105 leader_election.cc:288] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:34.482961 26163 leader_election.cc:288] T b4ebd2b043024a548005c8b6b00b0449 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
W20260925 03:04:34.484838 26121 consensus_meta.cc:287] T 43366a0a2f9d4344bf4e37ffd290ef8a P 869b0a8575b54d08ae2817fad5dd42d5: Time spent flushing consensus metadata: real 1.067s user 0.013s sys 0.003s
I20260925 03:04:34.463161 24465 raft_consensus.cc:2354] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 3. Current term is 4.
I20260925 03:04:34.463310 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "100c113a9fd84cc8adf2ec7b33ea0093" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:35.529528 25851 raft_consensus.cc:3032] T a4abe0ea757e4886be4945c5001e529a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 LEADER]: Stepping down as leader of term 1
I20260925 03:04:35.530026 25851 raft_consensus.cc:729] T a4abe0ea757e4886be4945c5001e529a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
W20260925 03:04:34.457692 24436 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.602s user 0.048s sys 0.070s
W20260925 03:04:34.490649 24540 consensus_meta.cc:287] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent flushing consensus metadata: real 0.506s user 0.008s sys 0.003s
I20260925 03:04:34.481272 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b030d84c670b48d5bff337ccc0c77626" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:34.430266 24539 raft_consensus.cc:2454] T b030d84c670b48d5bff337ccc0c77626 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 4.
I20260925 03:04:34.474426 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06f4a7eaf6534e4986c9fd35911038e4" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:35.543098 24368 catalog_manager.cc:5495] T 29637f5a4a504ce6ba29033ccd0cd716 P 1bc1f2e62ab44af995165bd5116a9d43 reported cstate change: config changed from index 2 to 3, 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) changed from NON_VOTER to VOTER, no longer has a pending config: VOTER 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1), VOTER a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2), VOTER eeab93abfdb141a895002ec9f312296c (127.22.193.3), VOTER 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4). New cstate: current_term: 3 leader_uuid: "eeab93abfdb141a895002ec9f312296c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:34.517547 26199 raft_consensus.cc:504] T 293ef69caf514ac98510a68495f54eda P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:35.545907 26199 leader_election.cc:288] T 293ef69caf514ac98510a68495f54eda P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:34.526904 26202 consensus_queue.cc:1035] T 3d89396576884f7c8901dba54f783947 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.010s
I20260925 03:04:34.528383 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b8e0d260a884ecdb20f79de46c36b19" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:34.492780 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74443a0a1dc34d88a2f42e3b722db405" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:35.551172 24613 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38564 (request call id 1210) took 1093 ms. Trace:
I20260925 03:04:35.551550 24613 rpcz_store.cc:276] 0925 03:04:34.457900 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:34.461285 (+ 3385us) service_pool.cc:227] Handling call
0925 03:04:35.551097 (+1089812us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:35.552675 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcf6e6a422d4d198d4fbb0f7d3d845d" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:35.553251 24613 raft_consensus.cc:2454] T edcf6e6a422d4d198d4fbb0f7d3d845d P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 1.
I20260925 03:04:35.553642 24613 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38564 (request call id 1215) took 1089 ms. Trace:
I20260925 03:04:35.553876 24613 rpcz_store.cc:276] 0925 03:04:34.464166 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:35.552338 (+1088172us) service_pool.cc:227] Handling call
0925 03:04:35.553598 (+ 1260us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:35.554816 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442d4a29b1d743599f4b6b4871e26668" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:35.555557 24613 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38564 (request call id 1213) took 1094 ms. Trace:
I20260925 03:04:35.555816 24613 rpcz_store.cc:276] 0925 03:04:34.460708 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:35.554497 (+1093789us) service_pool.cc:227] Handling call
0925 03:04:35.555515 (+ 1018us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:34.492429 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c88cbd39e01f44628a094af516be2833" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:35.557890 24617 raft_consensus.cc:2379] T c88cbd39e01f44628a094af516be2833 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 5: Already voted for candidate eeab93abfdb141a895002ec9f312296c in this term.
I20260925 03:04:35.558346 24617 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38564 (request call id 1211) took 1100 ms. Trace:
I20260925 03:04:35.558626 24617 rpcz_store.cc:276] 0925 03:04:34.458172 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:34.458292 (+ 120us) service_pool.cc:227] Handling call
0925 03:04:35.558282 (+1099990us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:35.559466 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5332a5d329493991dff994e70ea92c" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
W20260925 03:04:34.532648 25192 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49794 (request call id 148) took 2886 ms (client timeout 1857 ms). Trace:
W20260925 03:04:35.562668 25192 rpcz_store.cc:269] 0925 03:04:31.647582 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:31.647783 (+ 201us) service_pool.cc:227] Handling call
0925 03:04:34.532307 (+2884524us) spinlock_profiling.cc:243] Waited 48.6 ms on lock 0x7b54003c7dd4. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d42ae 00007f437b411c27 00007f4374af9c04 00007f4374af9b92 00007f4374af9ac1 00007f4374af9a3c 00007f4374af8cf2 00007f43748fa94c 00007f43748f9d86 00007f43748f972f
0925 03:04:34.532509 (+ 202us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":111743232}
I20260925 03:04:34.525238 25198 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49794 (request call id 154) took 1565 ms. Trace:
I20260925 03:04:34.496424 24614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e5819c15eac4544a57e5ee6c9d4f117" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:35.042903 24616 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34546 (request call id 79) took 2179 ms. Trace:
I20260925 03:04:35.564881 25198 rpcz_store.cc:276] 0925 03:04:32.960600 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:32.969692 (+ 9092us) service_pool.cc:227] Handling call
0925 03:04:33.000275 (+ 30583us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:33.283035 (+282760us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:33.283066 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:33.283106 (+ 40us) raft_consensus.cc:1631] Marking committed up to 5
0925 03:04:33.283135 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:34.524950 (+1241815us) spinlock_profiling.cc:243] Waited 283 ms on lock 0x7b54004bcdd4. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d0f51 00007f43764cf99e 00007f437b4112e9 00007f4374af7684 00007f4374af7612 00007f4374af7541 00007f4374af74bc 00007f4374af6772 00007f43748fa94c 00007f43748f9d86
0925 03:04:34.525010 (+ 60us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:34.525158 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":649769728}
I20260925 03:04:35.058801 24437 leader_election.cc:302] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:35.565692 24616 rpcz_store.cc:276] 0925 03:04:32.863819 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:32.867546 (+ 3727us) service_pool.cc:227] Handling call
0925 03:04:32.925373 (+ 57827us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:33.670245 (+744872us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:35.042575 (+1372330us) spinlock_profiling.cc:243] Waited 699 ms on lock 0x7b54000730d4. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d0f51 00007f43764cf99e 00007f437b4112e9 00007f4374af7684 00007f4374af7612 00007f4374af7541 00007f4374af74bc 00007f4374af6772 00007f43748fa94c 00007f43748f9d86
0925 03:04:35.042797 (+ 222us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1607192448}
W20260925 03:04:35.566390 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 507636us
I20260925 03:04:35.567118 24437 leader_election.cc:302] T c88cbd39e01f44628a094af516be2833 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:04:35.067732 24615 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38570 (request call id 1283) took 1207 ms. Trace:
I20260925 03:04:35.567346 24616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:35.070542 24620 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38570 (request call id 1284) took 1206 ms. Trace:
I20260925 03:04:35.088452 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bb46bce3564841b7674e6a693bd2e6" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:35.259423 26210 raft_consensus.cc:482] T a4abe0ea757e4886be4945c5001e529a P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:35.282145 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25102ba08e4e49388f6176badcd10a25" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:35.330960 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7691c6b475428c82d6181d693d26f3" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:34.568043 25804 raft_consensus.cc:482] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:35.571388 25804 raft_consensus.cc:504] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:35.574090 26175 raft_consensus.cc:482] T edcf6e6a422d4d198d4fbb0f7d3d845d P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:35.574609 26175 raft_consensus.cc:504] T edcf6e6a422d4d198d4fbb0f7d3d845d P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:34.520346 25897 raft_consensus.cc:482] T 43366a0a2f9d4344bf4e37ffd290ef8a P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:35.217507 25938 raft_consensus.cc:482] T 7993f9c3e2a44f408bbf5d1e395e2306 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:35.577505 25897 raft_consensus.cc:504] T 43366a0a2f9d4344bf4e37ffd290ef8a P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:34.987514 25805 leader_election.cc:288] T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:35.579945 24437 leader_election.cc:302] T 442d4a29b1d743599f4b6b4871e26668 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:04:34.475065 25973 consensus_queue.cc:237] T 7993f9c3e2a44f408bbf5d1e395e2306 P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:34.493276 25960 leader_election.cc:288] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
W20260925 03:04:34.461570 26065 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.555s user 0.047s sys 0.087s
I20260925 03:04:35.349676 24511 leader_election.cc:302] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
W20260925 03:04:35.588054 26065 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.689s user 0.051s sys 0.091s
I20260925 03:04:35.569293 24464 raft_consensus.cc:2396] T 11bb46bce3564841b7674e6a693bd2e6 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
W20260925 03:04:35.588490 24511 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.4:43979, user_credentials={real_user=slave}} blocked reactor thread for 238875us
I20260925 03:04:34.522934 26070 leader_election.cc:288] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:34.545660 26071 raft_consensus.cc:482] T cc2df40db08649c9a75e90000b878559 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:35.590181 26071 raft_consensus.cc:504] T cc2df40db08649c9a75e90000b878559 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:35.590852 26084 raft_consensus.cc:1050] T 367277daf8d04778b3fa4ed94d71580d P eeab93abfdb141a895002ec9f312296c: attempting to promote NON_VOTER 1bc1f2e62ab44af995165bd5116a9d43 to VOTER
I20260925 03:04:35.592195 26071 leader_election.cc:288] T cc2df40db08649c9a75e90000b878559 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:35.057832 26077 raft_consensus.cc:482] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:35.593406 26077 raft_consensus.cc:504] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
W20260925 03:04:34.503875 26113 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.694s user 0.005s sys 0.031s
I20260925 03:04:35.596840 24545 raft_consensus.cc:2454] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 4.
W20260925 03:04:35.094321 26115 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 2.251s user 0.012s sys 0.001s
I20260925 03:04:34.495708 26119 leader_election.cc:288] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:34.550406 26120 leader_election.cc:288] T 100c113a9fd84cc8adf2ec7b33ea0093 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 3 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
W20260925 03:04:34.505112 26125 consensus_meta.cc:287] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c: Time spent flushing consensus metadata: real 1.103s user 0.014s sys 0.002s
I20260925 03:04:35.602499 26125 raft_consensus.cc:504] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:35.603713 24617 raft_consensus.cc:2379] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 5: Already voted for candidate eeab93abfdb141a895002ec9f312296c in this term.
W20260925 03:04:34.549561 26126 consensus_meta.cc:287] T 37a157996e944db2906110d87ecb0c54 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent flushing consensus metadata: real 0.916s user 0.010s sys 0.004s
W20260925 03:04:34.993201 26128 log.cc:924] Time spent T 00000000000000000000000000000000 P 2d92cc0732c4490eb7d98d353826cce1: Append to log took a long time: real 2.357s user 0.006s sys 0.006s
I20260925 03:04:34.852433 26138 raft_consensus.cc:3032] T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 LEADER]: Stepping down as leader of term 2
I20260925 03:04:35.079923 26137 raft_consensus.cc:3032] T 5278a60f97da4e10aecd50a8d9a9a43c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 LEADER]: Stepping down as leader of term 1
I20260925 03:04:35.604571 26125 leader_election.cc:288] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
W20260925 03:04:35.083923 26141 log.cc:924] Time spent T b1e85248d3b04b979dd080caeeb02fdb P 1bc1f2e62ab44af995165bd5116a9d43: Append to log took a long time: real 2.259s user 0.000s sys 0.013s
W20260925 03:04:35.612529 24545 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 1235) took 2012 ms (client timeout 1861 ms). Trace:
I20260925 03:04:35.079082 26147 leader_election.cc:288] T b030d84c670b48d5bff337ccc0c77626 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
W20260925 03:04:34.547014 26171 consensus_meta.cc:287] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent flushing consensus metadata: real 0.617s user 0.011s sys 0.001s
I20260925 03:04:34.549790 26174 leader_election.cc:288] T 21c43af5e6844fdaacc8c625fa01fd8d P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:34.445756 26181 raft_consensus.cc:2781] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260925 03:04:35.617034 26181 raft_consensus.cc:482] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:35.617439 26181 raft_consensus.cc:3037] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:34.525116 26186 leader_election.cc:288] T b627ecb4a5384ef392aa695e06599c39 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:34.490183 26189 leader_election.cc:288] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:34.515379 26177 raft_consensus.cc:504] T 1414dcfe53f64a78ae413531eb8ec117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:34.470464 26191 raft_consensus.cc:482] T 71f461a6e2ff434993b78a2d077f0abd P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1bc1f2e62ab44af995165bd5116a9d43)
I20260925 03:04:34.563043 26193 leader_election.cc:288] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
W20260925 03:04:35.361258 24541 consensus_meta.cc:287] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent flushing consensus metadata: real 1.440s user 0.015s sys 0.011s
I20260925 03:04:35.623099 26191 raft_consensus.cc:504] T 71f461a6e2ff434993b78a2d077f0abd P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:35.623793 24541 raft_consensus.cc:2454] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 4.
W20260925 03:04:35.624378 24541 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 1238) took 2023 ms (client timeout 2007 ms). Trace:
W20260925 03:04:35.624770 24541 rpcz_store.cc:269] 0925 03:04:33.601199 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:33.759955 (+158756us) service_pool.cc:227] Handling call
0925 03:04:35.624304 (+1864349us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:35.625289 26191 leader_election.cc:288] T 71f461a6e2ff434993b78a2d077f0abd P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:35.625977 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9c33fd100ff44c1a3247319a324a044" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:35.626562 24541 raft_consensus.cc:2454] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 4.
I20260925 03:04:35.626955 24541 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36786 (request call id 690) took 2039 ms. Trace:
I20260925 03:04:35.627174 24541 rpcz_store.cc:276] 0925 03:04:33.587849 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:35.625693 (+2037844us) service_pool.cc:227] Handling call
0925 03:04:35.626906 (+ 1213us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:34.514789 26167 raft_consensus.cc:504] T 8ac466793d094325ace224708b2158ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
W20260925 03:04:35.368312 25163 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 178063us
W20260925 03:04:35.369318 25165 connection.cc:232] RPC call timeout handler was delayed by -1.2266490619986143s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.274s for a response
W20260925 03:04:34.473445 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 1.82075e+06us
I20260925 03:04:35.427832 24544 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 679) took 2533 ms. Trace:
I20260925 03:04:35.629985 26167 leader_election.cc:288] T 8ac466793d094325ace224708b2158ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:35.365157 24469 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55560 (request call id 1170) took 1097 ms. Trace:
W20260925 03:04:35.630525 24514 leader_election.cc:334] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187): Timed out: RequestConsensusVote RPC to 127.22.193.1:42187 timed out after 1.634s (SENT)
I20260925 03:04:35.630532 24469 rpcz_store.cc:276] 0925 03:04:34.267281 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:34.315075 (+ 47794us) service_pool.cc:227] Handling call
0925 03:04:34.400554 (+ 85479us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:35.364835 (+964281us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:35.365089 (+ 254us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:35.631213 25198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4ebd2b043024a548005c8b6b00b0449" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
W20260925 03:04:35.429141 24420 thread.cc:636] client-negotiator [worker] (thread pool) Time spent creating pthread: real 2.729s user 0.006s sys 0.013s
W20260925 03:04:35.632421 24420 thread.cc:602] client-negotiator [worker] (thread pool) Time spent starting thread: real 2.933s user 0.007s sys 0.013s
I20260925 03:04:35.429409 26086 raft_consensus.cc:3032] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 LEADER]: Stepping down as leader of term 3
I20260925 03:04:35.463829 24473 raft_consensus.cc:1226] T b627ecb4a5384ef392aa695e06599c39 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 3. Current term is 4. Ops: []
I20260925 03:04:35.632936 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f1fac028a144058b92331c0d9b44ebf" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:35.470068 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60d45b7d467d4e1ab92367f6d7d52b15" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:35.633293 24473 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55560 (request call id 1169) took 1366 ms. Trace:
W20260925 03:04:35.474056 24465 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1163) took 1377 ms (client timeout 1634 ms). Trace:
I20260925 03:04:35.502049 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "550118bc1a0641ce9803bdbf43f2069f" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:35.632571 25192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e068a0f80941dc9138cee063f53bab" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:35.634560 24546 raft_consensus.cc:3037] T 550118bc1a0641ce9803bdbf43f2069f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:35.508019 24470 raft_consensus.cc:3037] T a7aa47c48404465b94a36fd819fe3c7a P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:35.634752 25165 leader_election.cc:302] T 7e49248643df4131a1c42e74244e05c2 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:35.633636 24473 rpcz_store.cc:276] 0925 03:04:34.266920 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:34.267120 (+ 200us) service_pool.cc:227] Handling call
0925 03:04:35.463514 (+1196394us) spinlock_profiling.cc:243] Waited 43.3 ms on lock 0x7b5800001598. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f43733ecf11 00007f4373570851 00007f4373570702 00007f437c460b09 00007f437b39c730 00007f437b46279b 00007f437b462821 00007f437b4262b0 00007f437b4117f9 00007f437b4112a2 00007f4374af7684
0925 03:04:35.463692 (+ 178us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:35.633065 (+169373us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:35.633241 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":99483520}
I20260925 03:04:35.635732 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "550118bc1a0641ce9803bdbf43f2069f" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:35.636438 24473 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1171) took 1282 ms. Trace:
I20260925 03:04:35.634296 26086 raft_consensus.cc:729] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:04:35.637220 25190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4ebd2b043024a548005c8b6b00b0449" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:35.637717 26086 consensus_queue.cc:260] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:35.636374 25191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5aec38e66a947c681e6c2a72b05e6ca" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43"
I20260925 03:04:35.638697 26086 raft_consensus.cc:3037] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:35.639084 25191 raft_consensus.cc:2362] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 5. Re-sending same reply.
I20260925 03:04:35.511364 24619 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34546 (request call id 82) took 2026 ms. Trace:
I20260925 03:04:35.526430 24622 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38570 (request call id 1289) took 1625 ms. Trace:
I20260925 03:04:35.641131 25198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:35.641520 26076 raft_consensus.cc:2781] T 7e49248643df4131a1c42e74244e05c2 P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260925 03:04:35.641885 26076 raft_consensus.cc:482] T 7e49248643df4131a1c42e74244e05c2 P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:35.642202 26076 raft_consensus.cc:3037] T 7e49248643df4131a1c42e74244e05c2 P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:35.451344 24467 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 726) took 3127 ms. Trace:
I20260925 03:04:35.636821 25199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e54ca1757ee4e42956a9cebbeca4f62" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:35.643360 24467 rpcz_store.cc:276] 0925 03:04:32.323640 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:32.323807 (+ 167us) service_pool.cc:227] Handling call
0925 03:04:35.422569 (+3098762us) spinlock_profiling.cc:243] Waited 4.48 ms on lock 0x7b5800001598. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f43733ecf11 00007f4373570851 00007f4373570702 00007f437c460b09 00007f437b39c730 00007f437b46279b 00007f437b462821 00007f437b4262b0 00007f437b4117f9 00007f437b4112a2 00007f4374af7684
0925 03:04:35.422794 (+ 225us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:04:35.423195 (+ 401us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:35.423223 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:04:35.423693 (+ 470us) log.cc:842] Serialized 26 byte log entry
0925 03:04:35.449283 (+ 25590us) raft_consensus.cc:1631] Marking committed up to 2
0925 03:04:35.449327 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:35.449403 (+ 76us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:04:35.451006 (+ 1603us) raft_consensus.cc:1671] finished
0925 03:04:35.451026 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:35.451252 (+ 226us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1922,"raft.run_cpu_time_us":97,"raft.run_wall_time_us":99,"spinlock_wait_cycles":10303232,"thread_start_us":25229,"threads_started":1,"wal-append.queue_time_us":25605}
W20260925 03:04:35.537695 24436 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.687s user 0.050s sys 0.070s
W20260925 03:04:35.645053 24471 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 754) took 1190 ms (client timeout 1576 ms). Trace:
W20260925 03:04:35.645398 24471 rpcz_store.cc:269] 0925 03:04:34.455178 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:34.463914 (+ 8736us) service_pool.cc:227] Handling call
0925 03:04:35.644994 (+1181080us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:35.645727 25957 raft_consensus.cc:482] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:35.646270 25957 raft_consensus.cc:504] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:35.646657 26188 raft_consensus.cc:482] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:35.647361 25783 raft_consensus.cc:482] T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:35.647161 26188 raft_consensus.cc:504] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:35.647817 25783 raft_consensus.cc:504] T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:35.648231 25957 leader_election.cc:288] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:35.648943 25957 raft_consensus.cc:482] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:35.649174 26188 leader_election.cc:288] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:35.649339 25957 raft_consensus.cc:504] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:35.649695 25783 leader_election.cc:288] T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:35.646694 25741 raft_consensus.cc:482] T 0be4e0aa984f4f2b81a58e382e434ad5 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:35.531383 25851 consensus_queue.cc:260] T a4abe0ea757e4886be4945c5001e529a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:35.650724 25741 raft_consensus.cc:504] T 0be4e0aa984f4f2b81a58e382e434ad5 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:35.539809 25942 raft_consensus.cc:3037] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:35.640836 24619 rpcz_store.cc:276] 0925 03:04:33.484954 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:33.499700 (+ 14746us) service_pool.cc:227] Handling call
0925 03:04:33.510934 (+ 11234us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:04:33.540841 (+ 29907us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:33.540870 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:04:33.541371 (+ 501us) log.cc:842] Serialized 26 byte log entry
0925 03:04:33.598583 (+ 57212us) raft_consensus.cc:1631] Marking committed up to 4
0925 03:04:33.598611 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:35.511040 (+1912429us) spinlock_profiling.cc:243] Waited 29.5 ms on lock 0x7b54002ee5d4. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d0f51 00007f43764cf99e 00007f437b4112e9 00007f4374af7684 00007f4374af7612 00007f4374af7541 00007f4374af74bc 00007f4374af6772 00007f43748fa94c 00007f43748f9d86
0925 03:04:35.511078 (+ 38us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:04:35.511114 (+ 36us) raft_consensus.cc:1671] finished
0925 03:04:35.511121 (+ 7us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:35.511313 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":22705,"raft.queue_time_us":51436,"raft.run_cpu_time_us":309,"raft.run_wall_time_us":14870,"spinlock_wait_cycles":67917568,"thread_start_us":56788,"threads_started":1,"wal-append.queue_time_us":131784,"wal-append.run_cpu_time_us":31447,"wal-append.run_wall_time_us":1768605}
I20260925 03:04:35.652792 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b5df303cd6a4d59aca5c18bbbff24ed" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:35.653178 25741 leader_election.cc:288] T 0be4e0aa984f4f2b81a58e382e434ad5 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:35.653501 24619 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38564 (request call id 1219) took 1158 ms. Trace:
I20260925 03:04:35.540593 24540 raft_consensus.cc:2454] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 3.
W20260925 03:04:35.542356 24539 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 1244) took 1892 ms (client timeout 2100 ms). Trace:
I20260925 03:04:35.653851 24619 rpcz_store.cc:276] 0925 03:04:34.495232 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:35.652468 (+1157236us) service_pool.cc:227] Handling call
0925 03:04:35.653449 (+ 981us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260925 03:04:35.654068 24539 rpcz_store.cc:269] 0925 03:04:33.649743 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:34.368574 (+718831us) service_pool.cc:227] Handling call
0925 03:04:35.542296 (+1173722us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:35.655216 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c88cbd39e01f44628a094af516be2833" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:35.655802 24539 raft_consensus.cc:3037] T c88cbd39e01f44628a094af516be2833 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Advancing to term 5
W20260925 03:04:35.541747 24542 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 1249) took 1777 ms (client timeout 2178 ms). Trace:
I20260925 03:04:35.544114 24548 raft_consensus.cc:2454] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 4.
I20260925 03:04:34.507198 24547 raft_consensus.cc:2454] T 3d89396576884f7c8901dba54f783947 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 3.
W20260925 03:04:35.659351 24547 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 1253) took 1794 ms (client timeout 2144 ms). Trace:
W20260925 03:04:35.659986 24548 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 1255) took 1780 ms (client timeout 1999 ms). Trace:
I20260925 03:04:35.651294 25957 leader_election.cc:288] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
W20260925 03:04:35.634116 24465 rpcz_store.cc:269] 0925 03:04:34.096297 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:34.100599 (+ 4302us) service_pool.cc:227] Handling call
0925 03:04:35.473985 (+1373386us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:35.661401 24465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21c43af5e6844fdaacc8c625fa01fd8d" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:35.661566 25957 consensus_queue.cc:1035] T cc2df40db08649c9a75e90000b878559 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260925 03:04:35.660387 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:35.663349 24619 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34546 (request call id 84) took 1132 ms. Trace:
I20260925 03:04:35.663657 24619 rpcz_store.cc:276] 0925 03:04:34.530626 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:35.660038 (+1129412us) service_pool.cc:227] Handling call
0925 03:04:35.663290 (+ 3252us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:35.664485 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ac285175464fbda67211d9574db415" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:35.636788 24473 rpcz_store.cc:276] 0925 03:04:34.354493 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:35.635502 (+1281009us) service_pool.cc:227] Handling call
0925 03:04:35.636369 (+ 867us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:35.665243 24619 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38564 (request call id 1221) took 1169 ms. Trace:
I20260925 03:04:35.649699 26188 raft_consensus.cc:482] T e6b6ec010caf4689b8c90449764f5cf9 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1bc1f2e62ab44af995165bd5116a9d43)
I20260925 03:04:35.665616 24619 rpcz_store.cc:276] 0925 03:04:34.496050 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:35.664205 (+1168155us) service_pool.cc:227] Handling call
0925 03:04:35.665167 (+ 962us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:35.665951 26188 raft_consensus.cc:504] T e6b6ec010caf4689b8c90449764f5cf9 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:35.666780 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9c33fd100ff44c1a3247319a324a044" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:35.667366 24619 raft_consensus.cc:2454] T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 4.
I20260925 03:04:35.667804 24619 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38564 (request call id 1228) took 1135 ms. Trace:
I20260925 03:04:35.668056 24619 rpcz_store.cc:276] 0925 03:04:34.534767 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:35.666424 (+1131657us) service_pool.cc:227] Handling call
0925 03:04:35.667755 (+ 1331us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:35.653744 25741 consensus_queue.cc:1035] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.016s
I20260925 03:04:35.669410 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71f461a6e2ff434993b78a2d077f0abd" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:35.629526 24544 rpcz_store.cc:276] 0925 03:04:32.894852 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:33.575058 (+680206us) service_pool.cc:227] Handling call
0925 03:04:33.575356 (+ 298us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:04:33.658518 (+ 83162us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:33.658552 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:04:33.659123 (+ 571us) log.cc:842] Serialized 26 byte log entry
0925 03:04:35.022448 (+1363325us) spinlock_profiling.cc:243] Waited 3.73 ms on lock 0x7b1800043c00. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f437c4478b1 00007f437c454661 00007f437358a345 00007f437c3c1a39 00007f43735a78bb 00007f43735a5cbf 00007f43735a7e22 00007f43761515bb 00007f4376157063 00007f4376157411 00007f43764a45cf
0925 03:04:35.029213 (+ 6765us) raft_consensus.cc:1631] Marking committed up to 1
0925 03:04:35.029252 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:35.427501 (+398249us) spinlock_profiling.cc:243] Waited 78.3 ms on lock 0x7b54003b7754. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d0f51 00007f43764cf99e 00007f437b4112e9 00007f4374af7684 00007f4374af7612 00007f4374af7541 00007f4374af74bc 00007f4374af6772 00007f43748fa94c 00007f43748f9d86
0925 03:04:35.427554 (+ 53us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:04:35.427604 (+ 50us) raft_consensus.cc:1671] finished
0925 03:04:35.427615 (+ 11us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:35.427768 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4312,"raft.queue_time_us":8728,"raft.run_cpu_time_us":320,"raft.run_wall_time_us":24010,"spinlock_wait_cycles":188726528,"thread_start_us":6554,"threads_started":1,"wal-append.queue_time_us":1349177}
I20260925 03:04:35.670022 24619 raft_consensus.cc:2454] T 71f461a6e2ff434993b78a2d077f0abd P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 3.
W20260925 03:04:35.660308 24548 rpcz_store.cc:269] 0925 03:04:33.879398 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:34.461490 (+582092us) service_pool.cc:227] Handling call
0925 03:04:35.659925 (+1198435us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:35.670543 24619 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38564 (request call id 1222) took 1174 ms. Trace:
I20260925 03:04:35.641297 24622 rpcz_store.cc:276] 0925 03:04:33.901344 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:33.901489 (+ 145us) service_pool.cc:227] Handling call
0925 03:04:33.927328 (+ 25839us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:34.348063 (+420735us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:35.526163 (+1178100us) spinlock_profiling.cc:243] Waited 241 ms on lock 0x7b5400218ed4. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d0f51 00007f43764cf99e 00007f437b4112e9 00007f4374af7684 00007f4374af7612 00007f4374af7541 00007f4374af74bc 00007f4374af6772 00007f43748fa94c 00007f43748f9d86
0925 03:04:35.526366 (+ 203us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":554468736}
I20260925 03:04:35.670831 24619 rpcz_store.cc:276] 0925 03:04:34.496502 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:35.668657 (+1172155us) service_pool.cc:227] Handling call
0925 03:04:35.670497 (+ 1840us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:35.671577 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e5819c15eac4544a57e5ee6c9d4f117" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:35.671681 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e068a0f80941dc9138cee063f53bab" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:35.672166 24473 raft_consensus.cc:3037] T 8e5819c15eac4544a57e5ee6c9d4f117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:35.672276 24622 raft_consensus.cc:2454] T 04e068a0f80941dc9138cee063f53bab P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 2.
I20260925 03:04:35.672034 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be4e0aa984f4f2b81a58e382e434ad5" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:35.673256 24619 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38564 (request call id 1218) took 1178 ms. Trace:
W20260925 03:04:35.656881 24542 rpcz_store.cc:269] 0925 03:04:33.763902 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:34.470850 (+706948us) service_pool.cc:227] Handling call
0925 03:04:35.541676 (+1070826us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:35.673635 24619 rpcz_store.cc:276] 0925 03:04:34.494827 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:35.671692 (+1176865us) service_pool.cc:227] Handling call
0925 03:04:35.673205 (+ 1513us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:35.674396 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c23337a7cb74184a0e57b7095cb6a11" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:35.674935 24542 raft_consensus.cc:3037] T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:35.675335 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60d45b7d467d4e1ab92367f6d7d52b15" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:35.675935 24619 raft_consensus.cc:2454] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 4.
I20260925 03:04:35.677161 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b627ecb4a5384ef392aa695e06599c39" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:35.539634 24367 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39374 (request call id 204) took 6792 ms. Trace:
I20260925 03:04:35.567999 24620 rpcz_store.cc:276] 0925 03:04:33.864266 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:33.864390 (+ 124us) service_pool.cc:227] Handling call
0925 03:04:35.069786 (+1205396us) spinlock_profiling.cc:243] Waited 5.02 ms on lock 0x7b58001d3d98. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f43733ecf11 00007f4373570851 00007f4373570702 00007f437c460b09 00007f437b39c730 00007f437b46279b 00007f437b462821 00007f437b4262b0 00007f437b4117f9 00007f437b4112a2 00007f4374af7684
0925 03:04:35.070010 (+ 224us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:35.070185 (+ 175us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:35.070209 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:35.070248 (+ 39us) raft_consensus.cc:1631] Marking committed up to 3
0925 03:04:35.070274 (+ 26us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:35.070327 (+ 53us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:35.070453 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":11555712}
I20260925 03:04:35.569267 24468 raft_consensus.cc:2454] T 25102ba08e4e49388f6176badcd10a25 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 1.
I20260925 03:04:35.678648 24468 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1167) took 1429 ms. Trace:
I20260925 03:04:35.568914 26199 raft_consensus.cc:2726] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260925 03:04:35.569077 26210 raft_consensus.cc:504] T a4abe0ea757e4886be4945c5001e529a P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:35.569928 24466 raft_consensus.cc:2354] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 3. Current term is 5.
I20260925 03:04:35.670958 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60d45b7d467d4e1ab92367f6d7d52b15" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
W20260925 03:04:35.679546 24466 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1166) took 1458 ms (client timeout 1907 ms). Trace:
I20260925 03:04:35.679634 24548 raft_consensus.cc:2454] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 4.
I20260925 03:04:35.569864 26083 raft_consensus.cc:482] T 21c43af5e6844fdaacc8c625fa01fd8d P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:35.570392 26196 raft_consensus.cc:482] T 43366a0a2f9d4344bf4e37ffd290ef8a P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:35.680269 26083 raft_consensus.cc:504] T 21c43af5e6844fdaacc8c625fa01fd8d P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:35.680716 26210 leader_election.cc:288] T a4abe0ea757e4886be4945c5001e529a P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:35.680387 26196 raft_consensus.cc:504] T 43366a0a2f9d4344bf4e37ffd290ef8a P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:35.681010 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7aa47c48404465b94a36fd819fe3c7a" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:35.682344 26083 leader_election.cc:288] T 21c43af5e6844fdaacc8c625fa01fd8d P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:35.565857 24614 raft_consensus.cc:2354] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 2. Current term is 3.
I20260925 03:04:35.682953 26196 leader_election.cc:288] T 43366a0a2f9d4344bf4e37ffd290ef8a P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:35.683095 24614 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38564 (request call id 1209) took 1225 ms. Trace:
I20260925 03:04:35.683508 24614 rpcz_store.cc:276] 0925 03:04:34.457572 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:34.458557 (+ 985us) service_pool.cc:227] Handling call
0925 03:04:35.683036 (+1224479us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:35.578111 25938 raft_consensus.cc:504] T 7993f9c3e2a44f408bbf5d1e395e2306 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:35.684803 24614 raft_consensus.cc:1226] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260925 03:04:35.678819 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b030d84c670b48d5bff337ccc0c77626" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:35.685739 24620 raft_consensus.cc:2379] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 5: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:04:35.686779 24620 raft_consensus.cc:1226] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 3. Current term is 4. Ops: []
I20260925 03:04:35.687332 24620 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38564 (request call id 1225) took 1176 ms. Trace:
I20260925 03:04:35.678949 24468 rpcz_store.cc:276] 0925 03:04:34.249613 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:34.250404 (+ 791us) service_pool.cc:227] Handling call
0925 03:04:35.281929 (+1031525us) spinlock_profiling.cc:243] Waited 6.34 ms on lock 0x7b5800001598. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f43733ecf11 00007f4373570851 00007f4373570702 00007f437c460b09 00007f437b39c730 00007f437b46279b 00007f437b462821 00007f437b4121b0 00007f437b411a94 00007f4374af9c04 00007f4374af9b92
0925 03:04:35.678592 (+396663us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":14588672}
I20260925 03:04:35.668355 26188 leader_election.cc:288] T e6b6ec010caf4689b8c90449764f5cf9 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:35.689400 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74443a0a1dc34d88a2f42e3b722db405" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:35.580426 25804 leader_election.cc:288] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:35.584486 25960 raft_consensus.cc:2781] T c88cbd39e01f44628a094af516be2833 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260925 03:04:34.304354 26068 raft_consensus.cc:504] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:35.585320 25953 raft_consensus.cc:2781] T 442d4a29b1d743599f4b6b4871e26668 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260925 03:04:35.567622 24615 rpcz_store.cc:276] 0925 03:04:33.860291 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:33.860444 (+ 153us) service_pool.cc:227] Handling call
0925 03:04:35.067477 (+1207033us) spinlock_profiling.cc:243] Waited 606 ms on lock 0x7b54002b0a54. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d42ae 00007f437b411c27 00007f4374af9c04 00007f4374af9b92 00007f4374af9ac1 00007f4374af9a3c 00007f4374af8cf2 00007f43748fa94c 00007f43748f9d86 00007f43748f972f
0925 03:04:35.067633 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1393610112}
I20260925 03:04:35.691890 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7aa47c48404465b94a36fd819fe3c7a" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:35.692404 25953 raft_consensus.cc:482] T 442d4a29b1d743599f4b6b4871e26668 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:35.588604 25805 raft_consensus.cc:2781] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 03:04:35.588742 24464 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1168) took 1326 ms. Trace:
I20260925 03:04:35.576570 26175 leader_election.cc:288] T edcf6e6a422d4d198d4fbb0f7d3d845d P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:35.692227 24615 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34546 (request call id 85) took 1161 ms. Trace:
I20260925 03:04:35.589360 24585 leader_election.cc:302] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 1bc1f2e62ab44af995165bd5116a9d43, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:35.586905 25982 consensus_queue.cc:1046] T b1e85248d3b04b979dd080caeeb02fdb P 1bc1f2e62ab44af995165bd5116a9d43 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260925 03:04:35.585453 24437 leader_election.cc:302] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
W20260925 03:04:35.598942 26115 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 2.788s user 0.012s sys 0.002s
I20260925 03:04:35.605319 24617 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38564 (request call id 1214) took 1144 ms. Trace:
I20260925 03:04:35.693157 24615 rpcz_store.cc:276] 0925 03:04:34.530940 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:35.691451 (+1160511us) service_pool.cc:227] Handling call
0925 03:04:35.691736 (+ 285us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:35.691897 (+ 161us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:35.691919 (+ 22us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:35.691971 (+ 52us) raft_consensus.cc:1631] Marking committed up to 5
0925 03:04:35.692015 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:35.692066 (+ 51us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:35.692179 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:35.695333 26115 raft_consensus.cc:686] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [term 5 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:35.694060 25805 raft_consensus.cc:686] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 LEADER]: Becoming Leader. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:04:35.692417 26068 leader_election.cc:288] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:35.694168 25982 raft_consensus.cc:3032] T b1e85248d3b04b979dd080caeeb02fdb P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 LEADER]: Stepping down as leader of term 2
I20260925 03:04:35.608400 24543 raft_consensus.cc:2454] T 37a157996e944db2906110d87ecb0c54 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 4.
I20260925 03:04:35.695071 24539 raft_consensus.cc:2454] T c88cbd39e01f44628a094af516be2833 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 5.
I20260925 03:04:35.606561 24511 service_pool.cc:140] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:36802 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36786 (request call id 690)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 1254)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36786 (request call id 694)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36786 (request call id 699)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36786 (request call id 714)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36786 (request call id 708)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36786 (request call id 706)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36786 (request call id 701)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36786 (request call id 704)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36786 (request call id 697)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36786 (request call id 705)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36786 (request call id 696)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36786 (request call id 703)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 1258)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36786 (request call id 722)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36786 (request call id 710)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36786 (request call id 711)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36786 (request call id 717)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 1259)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36786 (request call id 698)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 1257)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36786 (request call id 725)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 1261)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36786 (request call id 715)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36786 (request call id 709)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36786 (request call id 724)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 680)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 682)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 684)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 686)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 688)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 691)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 1241)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 695)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 1243)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 1245)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 1247)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 1250)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 700)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 702)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 707)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 712)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 713)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 716)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 718)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 719)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 720)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 721)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 723)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 1256)
I20260925 03:04:35.693729 24469 raft_consensus.cc:2379] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 4: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:04:35.697130 25982 raft_consensus.cc:729] T b1e85248d3b04b979dd080caeeb02fdb P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1bc1f2e62ab44af995165bd5116a9d43, State: Running, Role: LEADER
W20260925 03:04:35.697990 24469 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1174) took 1253 ms (client timeout 1604 ms). Trace:
W20260925 03:04:35.698132 24543 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36786 (request call id 692) took 2101 ms (client timeout 1686 ms). Trace:
W20260925 03:04:35.698557 24543 rpcz_store.cc:269] 0925 03:04:33.597161 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:33.662050 (+ 64889us) service_pool.cc:227] Handling call
0925 03:04:35.697975 (+2035925us) spinlock_profiling.cc:243] Waited 1.92 s on lock 0x7b5400012554. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d42ae 00007f437b411c27 00007f4374af9c04 00007f4374af9b92 00007f4374af9ac1 00007f4374af9a3c 00007f4374af8cf2 00007f43748fa94c 00007f43748f9d86 00007f43748f972f
0925 03:04:35.698074 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4418387328}
I20260925 03:04:35.698369 25982 consensus_queue.cc:260] T b1e85248d3b04b979dd080caeeb02fdb P 1bc1f2e62ab44af995165bd5116a9d43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:35.693336 25953 raft_consensus.cc:3037] T 442d4a29b1d743599f4b6b4871e26668 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:35.699641 25982 raft_consensus.cc:3037] T b1e85248d3b04b979dd080caeeb02fdb P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:35.610613 26137 raft_consensus.cc:729] T 5278a60f97da4e10aecd50a8d9a9a43c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:04:35.695209 24617 rpcz_store.cc:276] 0925 03:04:34.461003 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:35.559216 (+1098213us) service_pool.cc:227] Handling call
0925 03:04:35.605113 (+ 45897us) spinlock_profiling.cc:243] Waited 43.5 ms on lock 0x7b54002eefd4. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d42ae 00007f437b411c27 00007f4374af9c04 00007f4374af9b92 00007f4374af9ac1 00007f4374af9a3c 00007f4374af8cf2 00007f43748fa94c 00007f43748f9d86 00007f43748f972f
0925 03:04:35.605251 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":100056320}
I20260925 03:04:35.700646 26137 consensus_queue.cc:260] T 5278a60f97da4e10aecd50a8d9a9a43c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:04:35.612962 24545 rpcz_store.cc:269] 0925 03:04:33.600162 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:33.668824 (+ 68662us) service_pool.cc:227] Handling call
0925 03:04:35.612339 (+1943515us) spinlock_profiling.cc:243] Waited 1.9 s on lock 0x7b54003b6d54. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d42ae 00007f437b411c27 00007f4374af9c04 00007f4374af9b92 00007f4374af9ac1 00007f4374af9a3c 00007f4374af8cf2 00007f43748fa94c 00007f43748f9d86 00007f43748f972f
0925 03:04:35.612463 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4377444096}
W20260925 03:04:35.694010 24585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.4:43979, user_credentials={real_user=slave}} blocked reactor thread for 104757us
I20260925 03:04:35.702528 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e5819c15eac4544a57e5ee6c9d4f117" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
W20260925 03:04:35.694871 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 109497us
I20260925 03:04:35.703769 24437 leader_election.cc:302] T 4e5332a5d329493991dff994e70ea92c P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:04:35.701871 26137 raft_consensus.cc:3037] T 5278a60f97da4e10aecd50a8d9a9a43c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:35.699817 26107 raft_consensus.cc:482] T 37a157996e944db2906110d87ecb0c54 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:35.705367 26107 raft_consensus.cc:504] T 37a157996e944db2906110d87ecb0c54 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:04:35.659736 24547 rpcz_store.cc:269] 0925 03:04:33.864911 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:34.503096 (+638185us) service_pool.cc:227] Handling call
0925 03:04:35.659284 (+1156188us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:35.706892 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74443a0a1dc34d88a2f42e3b722db405" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:35.707571 24547 raft_consensus.cc:2454] T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 4.
I20260925 03:04:35.595382 26077 leader_election.cc:288] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:35.623675 26177 leader_election.cc:288] T 1414dcfe53f64a78ae413531eb8ec117 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:35.628034 24586 leader_election.cc:302] T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
W20260925 03:04:35.631386 24514 leader_election.cc:334] T 0be4e0aa984f4f2b81a58e382e434ad5 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187): Timed out: RequestConsensusVote RPC to 127.22.193.1:42187 timed out after 1.638s (SENT)
I20260925 03:04:35.673306 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:35.634222 25194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c81a1b3b616492ab6c964f45af36077" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
W20260925 03:04:35.454761 24439 service_pool.cc:137] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:55562 dropped due to backpressure. The service queue is full; it has 50 items.
I20260925 03:04:35.531585 25195 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48424 (request call id 175) took 3309 ms. Trace:
I20260925 03:04:35.711249 25194 raft_consensus.cc:2454] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 3.
I20260925 03:04:35.711402 26177 raft_consensus.cc:2781] T 4e5332a5d329493991dff994e70ea92c P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader election won for term 4
W20260925 03:04:35.711839 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 80474.1us
I20260925 03:04:35.693280 24464 rpcz_store.cc:276] 0925 03:04:34.262052 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:34.262202 (+ 150us) service_pool.cc:227] Handling call
0925 03:04:35.088310 (+826108us) spinlock_profiling.cc:243] Waited 3.42 ms on lock 0x7b5800001598. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f43733ecf11 00007f4373570851 00007f4373570702 00007f437c460b09 00007f437b39c730 00007f437b46279b 00007f437b462821 00007f437b4121b0 00007f437b411a94 00007f4374af9c04 00007f4374af9b92
0925 03:04:35.588664 (+500354us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7863168}
W20260925 03:04:35.679927 24466 rpcz_store.cc:269] 0925 03:04:34.221141 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:34.226266 (+ 5125us) service_pool.cc:227] Handling call
0925 03:04:35.330727 (+1104461us) spinlock_profiling.cc:243] Waited 11.7 ms on lock 0x7b5800001598. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f43733ecf11 00007f4373570851 00007f4373570702 00007f437c460b09 00007f437b39c730 00007f437b46279b 00007f437b462821 00007f437b4121b0 00007f437b411a94 00007f4374af9c04 00007f4374af9b92
0925 03:04:35.679485 (+348758us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":26857728}
I20260925 03:04:35.709362 26199 raft_consensus.cc:2781] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260925 03:04:35.714686 26199 raft_consensus.cc:482] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:35.715297 26199 raft_consensus.cc:3037] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:35.706777 25953 raft_consensus.cc:504] T 442d4a29b1d743599f4b6b4871e26668 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:35.718389 26068 raft_consensus.cc:2781] T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260925 03:04:35.718885 25953 leader_election.cc:288] T 442d4a29b1d743599f4b6b4871e26668 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:35.718875 26068 raft_consensus.cc:482] T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260925 03:04:35.656937 24540 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 1236) took 2056 ms (client timeout 2070 ms). Trace:
W20260925 03:04:35.720718 24540 rpcz_store.cc:269] 0925 03:04:33.600562 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:33.858951 (+258389us) service_pool.cc:227] Handling call
0925 03:04:35.656856 (+1797905us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:35.721643 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ac285175464fbda67211d9574db415" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:35.722249 24540 raft_consensus.cc:2454] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 3.
I20260925 03:04:35.723212 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9357d144aa49789faaf31fe5e6bf89" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:35.723758 24540 raft_consensus.cc:2454] T ac9357d144aa49789faaf31fe5e6bf89 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 1.
I20260925 03:04:35.724810 26068 raft_consensus.cc:3037] T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:35.651572 25851 raft_consensus.cc:3037] T a4abe0ea757e4886be4945c5001e529a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:35.678073 24367 rpcz_store.cc:276] 0925 03:04:28.747292 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:28.750104 (+ 2812us) service_pool.cc:227] Handling call
0925 03:04:35.539563 (+6789459us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0925 03:04:31.039080 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0925 03:04:31.039678 (+ 598us) write_op.cc:432] Acquiring schema lock in shared mode
0925 03:04:31.039726 (+ 48us) write_op.cc:435] Acquired schema lock
0925 03:04:31.039742 (+ 16us) tablet.cc:660] Decoding operations
0925 03:04:31.040324 (+ 582us) write_op.cc:620] Acquiring the partition lock for write op
0925 03:04:31.040415 (+ 91us) write_op.cc:641] Partition lock acquired for write op
0925 03:04:31.040453 (+ 38us) tablet.cc:683] Acquiring locks for 4 operations
0925 03:04:31.041001 (+ 548us) tablet.cc:699] Row locks acquired
0925 03:04:31.041033 (+ 32us) write_op.cc:260] PREPARE: finished
0925 03:04:31.041172 (+ 139us) write_op.cc:270] Start()
0925 03:04:31.041301 (+ 129us) write_op.cc:276] Timestamp: P: 1790305471041121 usec, L: 0
0925 03:04:31.041320 (+ 19us) op_driver.cc:331] REPLICATION: starting
0925 03:04:31.041962 (+ 642us) log.cc:842] Serialized 1750 byte log entry
0925 03:04:31.318298 (+276336us) op_driver.cc:444] REPLICATION: finished
0925 03:04:31.581936 (+263638us) spinlock_profiling.cc:243] Waited 6.09 ms on lock 0x7b44000232cc. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f43733ecf11 00007f43734baa95 00007f43734bb4b2 00007f4376163511 00007f437615ee1a 00007f437358a5fe 00007f437c3c1a39 00007f43735a78bb 00007f43735a5cbf 00007f43735a7e22 00007f4379ee0410
0925 03:04:31.882040 (+300104us) write_op.cc:301] APPLY: starting
0925 03:04:31.884707 (+ 2667us) tablet.cc:1365] starting BulkCheckPresence
0925 03:04:31.915948 (+ 31241us) tablet.cc:1368] finished BulkCheckPresence
0925 03:04:31.919171 (+ 3223us) tablet.cc:1370] starting ApplyRowOperation cycle
0925 03:04:32.205349 (+286178us) tablet.cc:1381] finished ApplyRowOperation cycle
0925 03:04:32.216844 (+ 11495us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=4
0925 03:04:32.216894 (+ 50us) write_op.cc:312] APPLY: finished
0925 03:04:32.223041 (+ 6147us) log.cc:842] Serialized 40 byte log entry
0925 03:04:32.443022 (+219981us) write_op.cc:489] Releasing partition, row and schema locks
0925 03:04:32.460149 (+ 17127us) write_op.cc:454] Released schema lock
0925 03:04:32.463468 (+ 3319us) write_op.cc:341] FINISH: Updating metrics
0925 03:04:32.534824 (+ 71356us) write_op.cc:489] Releasing partition, row and schema locks
0925 03:04:32.544658 (+ 9834us) write_op.cc:454] Released schema lock
0925 03:04:34.992595 (+2447937us) spinlock_profiling.cc:243] Waited 1.48 ms on lock 0x7b44000232cc. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f43733ecf11 00007f43734baa95 00007f43734bb4b2 00007f4376163511 00007f437615ee1a 00007f4376152bee 00007f4376151f36 00007f4376151b58 00007f437615b551 00007f437615b509 00007f437615b499
Metrics: {"mutex_wait_us":67705,"reported_tablets":19,"child_traces":[["op",{"apply.queue_time_us":485929,"apply.run_cpu_time_us":87603,"apply.run_wall_time_us":686330,"num_ops":4,"prepare.queue_time_us":43614,"prepare.run_cpu_time_us":3258,"prepare.run_wall_time_us":19203,"replication_time_us":252085,"spinlock_wait_cycles":17418880,"thread_start_us":638907,"threads_started":3,"wal-append.queue_time_us":387215}]]}
I20260925 03:04:35.534619 24472 raft_consensus.cc:2454] T 100c113a9fd84cc8adf2ec7b33ea0093 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 2.
I20260925 03:04:35.704927 24437 leader_election.cc:302] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:35.729943 25953 raft_consensus.cc:2726] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260925 03:04:35.730755 25195 rpcz_store.cc:276] 0925 03:04:32.222308 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:32.224502 (+ 2194us) service_pool.cc:227] Handling call
0925 03:04:32.771975 (+547473us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:32.916372 (+144397us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:35.531341 (+2614969us) spinlock_profiling.cc:243] Waited 539 ms on lock 0x7b5400641750. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764cfa09 00007f437b4112e9 00007f4374af7684 00007f4374af7612 00007f4374af7541 00007f4374af74bc 00007f4374af6772 00007f43748fa94c 00007f43748f9d86 00007f43748f972f
0925 03:04:35.531500 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1239692032}
I20260925 03:04:35.732062 24622 raft_consensus.cc:2454] T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 3.
I20260925 03:04:35.732573 24622 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38564 (request call id 1224) took 1228 ms. Trace:
W20260925 03:04:35.732833 24472 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1173) took 1308 ms (client timeout 1726 ms). Trace:
I20260925 03:04:35.689519 25938 leader_election.cc:288] T 7993f9c3e2a44f408bbf5d1e395e2306 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:35.581076 25897 leader_election.cc:288] T 43366a0a2f9d4344bf4e37ffd290ef8a P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:35.690387 25960 raft_consensus.cc:482] T c88cbd39e01f44628a094af516be2833 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting leader election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:35.699554 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f22ddd7d1c444dc58f45a1fb0b3e02bc" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:35.733677 25960 raft_consensus.cc:3037] T c88cbd39e01f44628a094af516be2833 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:35.697343 25805 consensus_queue.cc:237] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:35.733871 24543 raft_consensus.cc:2454] T f22ddd7d1c444dc58f45a1fb0b3e02bc P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 4.
W20260925 03:04:35.711313 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 83379.5us
I20260925 03:04:35.734810 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceab6b0fccd54e909a64f66fe81a9a1f" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:35.712574 24439 service_pool.cc:140] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:55562 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 749)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1182)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1187)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1189)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 752)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1181)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1188)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 769)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 772)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1179)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1176)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1177)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1186)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1190)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 760)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 744)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 757)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 758)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 750)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 764)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 771)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 766)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 770)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 762)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 741)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 742)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 745)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 746)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 747)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55560 (request call id 1172)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55560 (request call id 1178)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55560 (request call id 1180)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55560 (request call id 1183)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55560 (request call id 1184)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55560 (request call id 1185)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55560 (request call id 1191)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 756)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 759)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 761)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 763)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 765)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 767)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 768)
Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:55552 (request call id 442)
I20260925 03:04:35.735451 24586 leader_election.cc:302] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:35.690344 24468 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 751) took 1268 ms. Trace:
I20260925 03:04:35.737191 26177 raft_consensus.cc:686] T 4e5332a5d329493991dff994e70ea92c P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 LEADER]: Becoming Leader. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:04:35.691093 24614 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38564 (request call id 1216) took 1220 ms. Trace:
I20260925 03:04:35.736770 24468 rpcz_store.cc:276] 0925 03:04:34.422032 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:35.688946 (+1266914us) service_pool.cc:227] Handling call
0925 03:04:35.690282 (+ 1336us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:35.739544 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "293ef69caf514ac98510a68495f54eda" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:35.740302 24468 raft_consensus.cc:2379] T 293ef69caf514ac98510a68495f54eda P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 3: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:04:35.692566 24618 raft_consensus.cc:2454] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 5.
I20260925 03:04:35.609869 26138 raft_consensus.cc:729] T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:04:35.740501 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9357d144aa49789faaf31fe5e6bf89" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:35.741276 24464 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 749) took 1326 ms. Trace:
I20260925 03:04:35.694238 26175 raft_consensus.cc:2726] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260925 03:04:35.742445 24618 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38570 (request call id 1297) took 1701 ms. Trace:
I20260925 03:04:35.742794 24618 rpcz_store.cc:276] 0925 03:04:34.041486 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:34.140956 (+ 99470us) service_pool.cc:227] Handling call
0925 03:04:35.742274 (+1601318us) spinlock_profiling.cc:243] Waited 1.35 s on lock 0x7b54003ac354. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d42ae 00007f437b411c27 00007f4374af9c04 00007f4374af9b92 00007f4374af9ac1 00007f4374af9a3c 00007f4374af8cf2 00007f43748fa94c 00007f43748f9d86 00007f43748f972f
0925 03:04:35.742396 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3111491840}
I20260925 03:04:35.743304 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7691c6b475428c82d6181d693d26f3" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:35.615177 26144 raft_consensus.cc:482] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:35.743760 24618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4abe0ea757e4886be4945c5001e529a" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:35.744204 24617 raft_consensus.cc:1261] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Refusing update from remote peer 1bc1f2e62ab44af995165bd5116a9d43: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260925 03:04:35.743970 26144 raft_consensus.cc:504] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:35.745317 24466 raft_consensus.cc:2379] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 5: Already voted for candidate eeab93abfdb141a895002ec9f312296c in this term.
I20260925 03:04:35.746080 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "420163a6cdce412ebc21c25fe3835638" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:35.746663 24466 raft_consensus.cc:2454] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 4.
I20260925 03:04:35.741640 26138 consensus_queue.cc:260] T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:35.747485 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "420163a6cdce412ebc21c25fe3835638" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:35.632040 25193 raft_consensus.cc:1226] T e92f377a79354a6ebb4eec67303c13e3 P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 LEADER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260925 03:04:35.747997 24586 leader_election.cc:302] T 37a157996e944db2906110d87ecb0c54 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:35.696439 26115 consensus_queue.cc:237] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:35.748934 24586 leader_election.cc:302] T 74443a0a1dc34d88a2f42e3b722db405 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:35.749709 24586 leader_election.cc:302] T ac9357d144aa49789faaf31fe5e6bf89 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:35.747596 26138 raft_consensus.cc:3037] T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:35.696331 24615 raft_consensus.cc:3037] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:35.619581 26084 consensus_queue.cc:237] T 367277daf8d04778b3fa4ed94d71580d P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 3, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:35.646353 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f22ddd7d1c444dc58f45a1fb0b3e02bc" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:35.681555 24548 raft_consensus.cc:2362] T a7aa47c48404465b94a36fd819fe3c7a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 2. Re-sending same reply.
I20260925 03:04:35.731899 24437 leader_election.cc:302] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:04:35.767290 24437 leader_election.cc:302] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:35.741575 24464 rpcz_store.cc:276] 0925 03:04:34.414880 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:35.740238 (+1325358us) service_pool.cc:227] Handling call
0925 03:04:35.741222 (+ 984us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:35.703177 24545 raft_consensus.cc:3037] T 8e5819c15eac4544a57e5ee6c9d4f117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Advancing to term 3
I20260925 03:04:35.769025 24543 raft_consensus.cc:2396] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 1.
I20260925 03:04:35.678275 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5f9b503b0ab43999d582015a367441e" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:35.657918 24467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b8e0d260a884ecdb20f79de46c36b19" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
W20260925 03:04:35.733289 24472 rpcz_store.cc:269] 0925 03:04:34.423987 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:34.443233 (+ 19246us) service_pool.cc:227] Handling call
0925 03:04:35.732775 (+1289542us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:35.687610 24620 rpcz_store.cc:276] 0925 03:04:34.511464 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:35.686252 (+1174788us) service_pool.cc:227] Handling call
0925 03:04:35.686600 (+ 348us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:35.687119 (+ 519us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:35.687280 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:35.738528 26177 consensus_queue.cc:237] T 4e5332a5d329493991dff994e70ea92c P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:35.796833 25938 raft_consensus.cc:2781] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260925 03:04:35.797302 25938 raft_consensus.cc:482] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:35.797643 25938 raft_consensus.cc:3037] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:35.754587 26120 raft_consensus.cc:2781] T 74443a0a1dc34d88a2f42e3b722db405 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260925 03:04:35.754973 26175 raft_consensus.cc:2781] T 37a157996e944db2906110d87ecb0c54 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260925 03:04:35.754990 25193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e54ca1757ee4e42956a9cebbeca4f62" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:35.736872 26196 raft_consensus.cc:2781] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260925 03:04:35.756018 26083 raft_consensus.cc:2781] T ac9357d144aa49789faaf31fe5e6bf89 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 03:04:35.756613 25960 raft_consensus.cc:504] T c88cbd39e01f44628a094af516be2833 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:35.813071 24546 raft_consensus.cc:2454] T 550118bc1a0641ce9803bdbf43f2069f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 3.
I20260925 03:04:35.813194 25960 leader_election.cc:288] T c88cbd39e01f44628a094af516be2833 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
W20260925 03:04:35.757774 24514 connection.cc:232] RPC call timeout handler was delayed by -3.0435594179998589s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18580000000000002s for a response
W20260925 03:04:35.814021 24514 connection.cc:232] RPC call timeout handler was delayed by -2.999459418000697s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1907s for a response
W20260925 03:04:35.814319 24514 connection.cc:232] RPC call timeout handler was delayed by -2.9292594180005835s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1985s for a response
W20260925 03:04:35.814575 24514 connection.cc:232] RPC call timeout handler was delayed by -2.9112594180005544s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2005s for a response
W20260925 03:04:35.814821 24514 connection.cc:232] RPC call timeout handler was delayed by -2.9013594180014479s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2016s for a response
I20260925 03:04:35.815117 24470 raft_consensus.cc:2454] T a7aa47c48404465b94a36fd819fe3c7a P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 2.
W20260925 03:04:35.815637 25900 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 2 current term: 2 is higher.
W20260925 03:04:35.816061 24514 connection.cc:232] RPC call timeout handler was delayed by -2.879759418001413s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20400000000000001s for a response
I20260925 03:04:35.758661 26144 leader_election.cc:288] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:35.816636 25804 raft_consensus.cc:482] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:35.817189 25804 raft_consensus.cc:504] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:35.760633 24618 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1d7691c6b475428c82d6181d693d26f3"
dest_uuid: "eeab93abfdb141a895002ec9f312296c"
mode: GRACEFUL
new_leader_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
from {username='slave'} at 127.0.0.1:44570
I20260925 03:04:35.818142 24618 raft_consensus.cc:595] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [term 5 LEADER]: Received request to transfer leadership to TS a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:35.815604 24437 service_pool.cc:140] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:55560 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 772)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1179)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1176)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1177)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1186)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1190)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 760)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 744)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 757)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1193)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 758)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1199)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 750)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 764)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 780)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1196)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 771)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1197)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1200)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 766)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1195)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1192)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1194)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 770)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 762)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 741)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 742)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 745)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 746)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 747)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55560 (request call id 1172)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55560 (request call id 1178)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55560 (request call id 1180)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55560 (request call id 1183)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55560 (request call id 1184)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55560 (request call id 1185)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55560 (request call id 1191)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 756)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 759)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 761)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 763)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 765)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 767)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 768)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 774)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 775)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 776)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 777)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 778)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 779)
I20260925 03:04:35.756597 24586 leader_election.cc:302] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
W20260925 03:04:35.698386 24469 rpcz_store.cc:269] 0925 03:04:34.444744 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:35.632654 (+1187910us) service_pool.cc:227] Handling call
0925 03:04:35.697931 (+ 65277us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260925 03:04:35.819334 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 62819.6us
I20260925 03:04:35.820150 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7aa47c48404465b94a36fd819fe3c7a" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:35.763465 25196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71f461a6e2ff434993b78a2d077f0abd" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:35.694504 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71f461a6e2ff434993b78a2d077f0abd" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:35.746361 24614 rpcz_store.cc:276] 0925 03:04:34.470619 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:35.684130 (+1213511us) service_pool.cc:227] Handling call
0925 03:04:35.684514 (+ 384us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:04:35.690838 (+ 6324us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:35.691041 (+ 203us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:35.556536 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f22ddd7d1c444dc58f45a1fb0b3e02bc" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:35.768334 24471 raft_consensus.cc:2454] T f22ddd7d1c444dc58f45a1fb0b3e02bc P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 4.
I20260925 03:04:35.822856 24471 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 753) took 1374 ms. Trace:
I20260925 03:04:35.823242 24471 rpcz_store.cc:276] 0925 03:04:34.448901 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:35.646102 (+1197201us) service_pool.cc:227] Handling call
0925 03:04:35.822780 (+176678us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260925 03:04:35.816562 24514 connection.cc:232] RPC call timeout handler was delayed by -2.8410594180004409s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20830000000000004s for a response
I20260925 03:04:35.819919 24437 service_pool.cc:140] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:55562 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1179)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1176)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1177)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1186)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1190)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 760)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 744)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 757)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1193)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 758)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1199)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 750)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 764)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 780)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1196)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 771)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1197)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1200)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 766)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1201)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1195)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1192)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1194)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 770)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 762)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 741)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 742)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 745)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 746)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 747)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55560 (request call id 1172)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55560 (request call id 1178)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55560 (request call id 1180)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55560 (request call id 1183)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55560 (request call id 1184)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55560 (request call id 1185)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55560 (request call id 1191)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 756)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 759)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 761)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 763)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 765)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 767)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 768)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 774)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 775)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 776)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 777)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 778)
W20260925 03:04:35.823887 24514 connection.cc:232] RPC call timeout handler was delayed by -2.8230594180004118s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21030000000000004s for a response
I20260925 03:04:35.732892 24622 rpcz_store.cc:276] 0925 03:04:34.504185 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:35.673046 (+1168861us) service_pool.cc:227] Handling call
0925 03:04:35.732519 (+ 59473us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:35.724612 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f1fac028a144058b92331c0d9b44ebf" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:35.708359 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ac466793d094325ace224708b2158ed" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:35.786013 24467 raft_consensus.cc:2454] T 7b8e0d260a884ecdb20f79de46c36b19 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 3.
I20260925 03:04:35.825985 24467 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 743) took 1491 ms. Trace:
I20260925 03:04:35.794338 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c85f5d1d3f004cf5817bb594fd8e6469" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:35.548110 24368 catalog_manager.cc:5495] T 04e068a0f80941dc9138cee063f53bab P 1bc1f2e62ab44af995165bd5116a9d43 reported cstate change: term changed from 1 to 2, leader changed from a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2) to 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4). New cstate: current_term: 2 leader_uuid: "1bc1f2e62ab44af995165bd5116a9d43" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260925 03:04:35.850485 24437 service_pool.cc:140] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:55562 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1177)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1186)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1190)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 760)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 744)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 757)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1193)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 758)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1203)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1199)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 750)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 764)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 780)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1196)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 771)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1197)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1200)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 766)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1201)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1195)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1204)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1192)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1194)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1202)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 770)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 762)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 741)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 742)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 745)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 746)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 747)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55560 (request call id 1172)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55560 (request call id 1178)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55560 (request call id 1180)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55560 (request call id 1183)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55560 (request call id 1184)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55560 (request call id 1185)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55560 (request call id 1191)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 756)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 759)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 761)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 763)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 765)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 767)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 768)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 774)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 775)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 776)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 777)
I20260925 03:04:35.738504 24439 service_pool.cc:140] GetConsensusState request on kudu.consensus.ConsensusService from 127.0.0.1:55552 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1176)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1177)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1186)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1190)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 760)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 744)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 757)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1193)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 758)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1203)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1199)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 750)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 764)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 780)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1196)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 771)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1197)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1200)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 766)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1201)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1195)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1204)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1192)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1194)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1202)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 770)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 762)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 741)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 742)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 745)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 746)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 747)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55560 (request call id 1172)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55560 (request call id 1178)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55560 (request call id 1180)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55560 (request call id 1183)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55560 (request call id 1184)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55560 (request call id 1185)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55560 (request call id 1191)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 756)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 759)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 761)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 763)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 765)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 767)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 768)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 774)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 775)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 776)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 777)
I20260925 03:04:35.827497 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c23337a7cb74184a0e57b7095cb6a11" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:35.839425 24465 raft_consensus.cc:2454] T 21c43af5e6844fdaacc8c625fa01fd8d P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 2.
I20260925 03:04:35.748025 24466 raft_consensus.cc:2454] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 4.
W20260925 03:04:38.180799 24466 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 752) took 3743 ms (client timeout 2446 ms). Trace:
W20260925 03:04:38.181057 24466 rpcz_store.cc:269] 0925 03:04:34.437678 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:35.747255 (+1309577us) service_pool.cc:227] Handling call
0925 03:04:38.180722 (+2433467us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260925 03:04:38.181775 24466 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1177) took 3157 ms (client timeout 2201 ms). Trace:
W20260925 03:04:38.182042 24466 rpcz_store.cc:269] 0925 03:04:35.024638 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:38.181623 (+3156985us) service_pool.cc:214] Skipping call since client already timed out
0925 03:04:38.181721 (+ 98us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20260925 03:04:38.182549 24466 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1186) took 3155 ms (client timeout 2200 ms). Trace:
W20260925 03:04:38.182726 24466 rpcz_store.cc:269] 0925 03:04:35.027388 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:38.182429 (+3155041us) service_pool.cc:214] Skipping call since client already timed out
0925 03:04:38.182498 (+ 69us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20260925 03:04:38.183136 24466 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1190) took 3154 ms (client timeout 2206 ms). Trace:
W20260925 03:04:38.183302 24466 rpcz_store.cc:269] 0925 03:04:35.028428 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:38.183052 (+3154624us) service_pool.cc:214] Skipping call since client already timed out
0925 03:04:38.183112 (+ 60us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20260925 03:04:38.183732 24466 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 760) took 2735 ms (client timeout 1801 ms). Trace:
W20260925 03:04:38.183913 24466 rpcz_store.cc:269] 0925 03:04:35.448063 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:38.183645 (+2735582us) service_pool.cc:214] Skipping call since client already timed out
0925 03:04:38.183707 (+ 62us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20260925 03:04:38.184389 24466 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 744) took 3845 ms (client timeout 2938 ms). Trace:
I20260925 03:04:35.826727 24467 rpcz_store.cc:276] 0925 03:04:34.335108 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:35.657583 (+1322475us) service_pool.cc:227] Handling call
0925 03:04:35.825921 (+168338us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260925 03:04:35.816341 24470 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 710) took 4323 ms (client timeout 1630 ms). Trace:
I20260925 03:04:35.837376 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43366a0a2f9d4344bf4e37ffd290ef8a" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:38.191514 24471 raft_consensus.cc:2454] T 43366a0a2f9d4344bf4e37ffd290ef8a P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 2.
I20260925 03:04:35.804713 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c23337a7cb74184a0e57b7095cb6a11" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:35.801362 24473 raft_consensus.cc:2454] T 8e5819c15eac4544a57e5ee6c9d4f117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 3.
I20260925 03:04:35.835984 24511 service_pool.cc:140] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:36802 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 1266)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36786 (request call id 710)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 1264)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 1271)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 1273)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 1274)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 1272)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36786 (request call id 711)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 1267)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36786 (request call id 728)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36786 (request call id 717)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 1259)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36786 (request call id 698)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 1257)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36786 (request call id 727)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36786 (request call id 725)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 1261)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 1270)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 1265)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 1263)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36786 (request call id 715)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36786 (request call id 709)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36786 (request call id 724)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 1275)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 680)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 682)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 684)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 686)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 688)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 691)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 1241)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 695)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 1243)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 1245)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 1247)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 1250)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 700)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 702)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 707)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 712)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 713)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 716)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 718)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 719)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 720)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 721)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 723)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 1256)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 729)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 1268)
W20260925 03:04:35.824977 24514 connection.cc:232] RPC call timeout handler was delayed by -2.7348594180002692s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22010000000000002s for a response
I20260925 03:04:38.194761 24511 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 1268) took 2361 ms. Trace:
I20260925 03:04:38.195086 24511 rpcz_store.cc:276] 0925 03:04:35.833045 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:38.194707 (+2361662us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260925 03:04:35.699642 24539 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36786 (request call id 694) took 2076 ms. Trace:
I20260925 03:04:38.198125 24539 rpcz_store.cc:276] 0925 03:04:33.623721 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:35.654915 (+2031194us) service_pool.cc:227] Handling call
0925 03:04:35.699585 (+ 44670us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:36.039155 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21c43af5e6844fdaacc8c625fa01fd8d" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:35.627976 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5278a60f97da4e10aecd50a8d9a9a43c" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:35.880534 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c85f5d1d3f004cf5817bb594fd8e6469" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:38.200378 24541 raft_consensus.cc:2454] T 5278a60f97da4e10aecd50a8d9a9a43c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 2.
I20260925 03:04:38.200834 24543 raft_consensus.cc:2454] T c85f5d1d3f004cf5817bb594fd8e6469 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 3.
W20260925 03:04:38.201040 24541 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 1254) took 4326 ms (client timeout 2818 ms). Trace:
W20260925 03:04:35.813657 24546 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 1240) took 2207 ms (client timeout 1979 ms). Trace:
W20260925 03:04:38.203879 24546 rpcz_store.cc:269] 0925 03:04:33.619020 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:33.762736 (+143716us) service_pool.cc:227] Handling call
0925 03:04:35.501889 (+1739153us) spinlock_profiling.cc:243] Waited 5.83 ms on lock 0x7b5800003c98. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f43733ecf11 00007f4373570851 00007f4373570702 00007f437c460b09 00007f437b39c730 00007f437b46279b 00007f437b462821 00007f437b4121b0 00007f437b411a94 00007f4374af9c04 00007f4374af9b92
0925 03:04:35.813592 (+311703us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":13398912}
W20260925 03:04:38.204818 24546 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 1271) took 2370 ms (client timeout 1758 ms). Trace:
W20260925 03:04:38.205099 24546 rpcz_store.cc:269] 0925 03:04:35.834472 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:38.204634 (+2370162us) service_pool.cc:214] Skipping call since client already timed out
0925 03:04:38.204767 (+ 133us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20260925 03:04:38.205742 24546 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 1273) took 2370 ms (client timeout 1776 ms). Trace:
W20260925 03:04:38.206009 24546 rpcz_store.cc:269] 0925 03:04:35.835046 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:38.205615 (+2370569us) service_pool.cc:214] Skipping call since client already timed out
0925 03:04:38.205703 (+ 88us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20260925 03:04:38.206633 24546 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 1274) took 2371 ms (client timeout 1802 ms). Trace:
W20260925 03:04:38.206898 24546 rpcz_store.cc:269] 0925 03:04:35.835327 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:38.206507 (+2371180us) service_pool.cc:214] Skipping call since client already timed out
0925 03:04:38.206592 (+ 85us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20260925 03:04:38.207515 24546 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 1272) took 2372 ms (client timeout 1820 ms). Trace:
W20260925 03:04:38.207780 24546 rpcz_store.cc:269] 0925 03:04:35.834770 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:38.207382 (+2372612us) service_pool.cc:214] Skipping call since client already timed out
0925 03:04:38.207467 (+ 85us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20260925 03:04:38.208439 24546 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36786 (request call id 711) took 2821 ms (client timeout 2281 ms). Trace:
I20260925 03:04:35.839051 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcf6e6a422d4d198d4fbb0f7d3d845d" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:35.889046 24542 raft_consensus.cc:2454] T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 5.
W20260925 03:04:38.211287 24542 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36786 (request call id 714) took 2823 ms (client timeout 1626 ms). Trace:
W20260925 03:04:38.211607 24542 rpcz_store.cc:269] 0925 03:04:35.387545 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:35.674129 (+286584us) service_pool.cc:227] Handling call
0925 03:04:38.211223 (+2537094us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260925 03:04:38.212342 24542 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 1267) took 2379 ms (client timeout 1867 ms). Trace:
W20260925 03:04:38.212646 24542 rpcz_store.cc:269] 0925 03:04:35.832580 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:38.212200 (+2379620us) service_pool.cc:214] Skipping call since client already timed out
0925 03:04:38.212295 (+ 95us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20260925 03:04:38.213296 24542 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36786 (request call id 728) took 2477 ms (client timeout 2031 ms). Trace:
W20260925 03:04:38.213570 24542 rpcz_store.cc:269] 0925 03:04:35.736223 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:38.213170 (+2476947us) service_pool.cc:214] Skipping call since client already timed out
0925 03:04:38.213257 (+ 87us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20260925 03:04:38.214166 24542 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36786 (request call id 717) took 2640 ms (client timeout 2242 ms). Trace:
W20260925 03:04:38.214425 24542 rpcz_store.cc:269] 0925 03:04:35.573294 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:38.214044 (+2640750us) service_pool.cc:214] Skipping call since client already timed out
0925 03:04:38.214131 (+ 87us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20260925 03:04:38.215044 24542 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 1259) took 2610 ms (client timeout 2218 ms). Trace:
W20260925 03:04:38.215307 24542 rpcz_store.cc:269] 0925 03:04:35.605416 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:38.214920 (+2609504us) service_pool.cc:214] Skipping call since client already timed out
0925 03:04:38.215008 (+ 88us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20260925 03:04:38.215929 24542 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36786 (request call id 698) took 2836 ms (client timeout 2446 ms). Trace:
W20260925 03:04:38.216189 24542 rpcz_store.cc:269] 0925 03:04:35.379392 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:38.215807 (+2836415us) service_pool.cc:214] Skipping call since client already timed out
0925 03:04:38.215893 (+ 86us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260925 03:04:35.884231 24545 raft_consensus.cc:2454] T 8e5819c15eac4544a57e5ee6c9d4f117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 3.
W20260925 03:04:38.217382 24545 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36786 (request call id 704) took 2834 ms (client timeout 2028 ms). Trace:
W20260925 03:04:38.217715 24545 rpcz_store.cc:269] 0925 03:04:35.383055 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:35.702248 (+319193us) service_pool.cc:227] Handling call
0925 03:04:38.217320 (+2515072us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260925 03:04:38.218408 24545 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36786 (request call id 727) took 2482 ms (client timeout 2113 ms). Trace:
W20260925 03:04:38.218712 24545 rpcz_store.cc:269] 0925 03:04:35.735938 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:38.218268 (+2482330us) service_pool.cc:214] Skipping call since client already timed out
0925 03:04:38.218363 (+ 95us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20260925 03:04:38.219326 24545 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36786 (request call id 725) took 2614 ms (client timeout 2558 ms). Trace:
W20260925 03:04:38.219597 24545 rpcz_store.cc:269] 0925 03:04:35.605292 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:38.219201 (+2613909us) service_pool.cc:214] Skipping call since client already timed out
0925 03:04:38.219285 (+ 84us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20260925 03:04:36.733987 26230 log.cc:924] Time spent T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.171s user 0.001s sys 0.000s
I20260925 03:04:36.801390 26239 raft_consensus.cc:482] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:35.820410 24586 leader_election.cc:302] T a7aa47c48404465b94a36fd819fe3c7a P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
W20260925 03:04:35.909991 24585 leader_election.cc:334] T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979): Timed out: RequestConsensusVote RPC to 127.22.193.4:43979 timed out after 1.857s (SENT)
I20260925 03:04:38.228039 24585 leader_election.cc:302] T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters: 1bc1f2e62ab44af995165bd5116a9d43, a04a37fc4aa7460cbfdbeeb4e451e1f5
W20260925 03:04:36.257308 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 346215us
W20260925 03:04:38.230409 24588 connection.cc:232] RPC call timeout handler was delayed by -2.3039097010005207s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1801s for a response
I20260925 03:04:35.850184 24613 raft_consensus.cc:2396] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260925 03:04:38.232812 26115 raft_consensus.cc:2781] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260925 03:04:38.232901 24613 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38564 (request call id 1226) took 3716 ms (client timeout 1824 ms). Trace:
W20260925 03:04:38.233385 24613 rpcz_store.cc:269] 0925 03:04:34.516787 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:35.556329 (+1039542us) service_pool.cc:227] Handling call
0925 03:04:38.232802 (+2676473us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:35.890795 24615 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34546 (request call id 86) took 1268 ms. Trace:
W20260925 03:04:36.710475 25165 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.516s user 0.041s sys 0.083s
W20260925 03:04:38.238560 25165 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 2.044s user 0.041s sys 0.085s
W20260925 03:04:38.239414 25165 connection.cc:232] RPC call timeout handler was delayed by -1.0992242299998907s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19690000000000002s for a response
I20260925 03:04:35.806137 24619 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38564 (request call id 1227) took 1277 ms. Trace:
I20260925 03:04:36.758167 25194 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49794 (request call id 162) took 1031 ms. Trace:
I20260925 03:04:38.241952 25982 raft_consensus.cc:482] T b1e85248d3b04b979dd080caeeb02fdb P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:35.819249 25804 leader_election.cc:288] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:38.244601 25804 raft_consensus.cc:482] T 48ca6ccd98194cf2af7b5278d15bf4ea P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:38.245093 25804 raft_consensus.cc:504] T 48ca6ccd98194cf2af7b5278d15bf4ea P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:38.247053 25804 leader_election.cc:288] T 48ca6ccd98194cf2af7b5278d15bf4ea P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:35.799268 25897 raft_consensus.cc:2726] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260925 03:04:36.249621 25942 consensus_meta.cc:287] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c: Time spent flushing consensus metadata: real 0.598s user 0.012s sys 0.001s
I20260925 03:04:35.836680 25938 raft_consensus.cc:504] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:38.253072 25938 leader_election.cc:288] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:35.830086 26068 raft_consensus.cc:504] T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:04:36.250748 26076 consensus_meta.cc:287] T 7e49248643df4131a1c42e74244e05c2 P 1bc1f2e62ab44af995165bd5116a9d43: Time spent flushing consensus metadata: real 0.608s user 0.008s sys 0.002s
I20260925 03:04:35.835922 26077 consensus_queue.cc:1035] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260925 03:04:38.220332 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35992316cb8d4c028333388eff02ecde" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:35.811586 26083 raft_consensus.cc:482] T ac9357d144aa49789faaf31fe5e6bf89 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting leader election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:38.258836 26083 raft_consensus.cc:3037] T ac9357d144aa49789faaf31fe5e6bf89 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Advancing to term 2
W20260925 03:04:38.262143 24546 rpcz_store.cc:269] 0925 03:04:35.386654 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:38.208306 (+2821652us) service_pool.cc:214] Skipping call since client already timed out
0925 03:04:38.208400 (+ 94us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260925 03:04:38.263398 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "293ef69caf514ac98510a68495f54eda" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
W20260925 03:04:38.264253 24546 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 1270) took 2430 ms (client timeout 2566 ms). Trace:
I20260925 03:04:36.046646 26107 leader_election.cc:288] T 37a157996e944db2906110d87ecb0c54 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
W20260925 03:04:36.755692 26084 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.513s user 0.045s sys 0.021s
W20260925 03:04:38.267729 26084 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.030s user 0.047s sys 0.022s
I20260925 03:04:35.804147 26120 raft_consensus.cc:482] T 74443a0a1dc34d88a2f42e3b722db405 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting leader election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
W20260925 03:04:36.562983 26138 consensus_meta.cc:287] T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent flushing consensus metadata: real 0.813s user 0.010s sys 0.005s
I20260925 03:04:36.090229 26150 raft_consensus.cc:482] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:38.195746 24511 service_pool.cc:140] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:36802 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 1264)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 1271)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 1273)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 1274)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 1272)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36786 (request call id 711)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 1267)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36786 (request call id 728)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36786 (request call id 717)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 1259)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36786 (request call id 698)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 1257)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36786 (request call id 727)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36786 (request call id 725)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 1261)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 1270)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 1265)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 1263)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36786 (request call id 715)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36786 (request call id 709)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36786 (request call id 724)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 1275)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 1278)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 1276)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 1277)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 680)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 682)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 684)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 686)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 688)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 691)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 1241)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 695)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 1243)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 1245)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 1247)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 1250)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 700)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 702)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 707)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 712)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 713)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 716)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 718)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 719)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 720)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 721)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 723)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 1256)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 729)
I20260925 03:04:38.270524 26150 raft_consensus.cc:504] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:35.807753 26175 raft_consensus.cc:482] T 37a157996e944db2906110d87ecb0c54 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:35.831808 26181 raft_consensus.cc:504] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:35.811401 26196 raft_consensus.cc:482] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:35.833951 26199 raft_consensus.cc:504] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:04:38.276881 24542 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 1257) took 2673 ms (client timeout 2240 ms). Trace:
W20260925 03:04:38.277269 24542 rpcz_store.cc:269] 0925 03:04:35.603635 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:38.216675 (+2613040us) service_pool.cc:214] Skipping call since client already timed out
0925 03:04:38.276813 (+ 60138us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260925 03:04:38.278016 26199 leader_election.cc:288] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:35.821391 26209 raft_consensus.cc:482] T a7aa47c48404465b94a36fd819fe3c7a P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:37.671221 26240 raft_consensus.cc:482] T a4abe0ea757e4886be4945c5001e529a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260925 03:04:37.847674 24569 debug-util.cc:398] Leaking SignalData structure 0x7b0800194e60 after lost signal to thread 23300
I20260925 03:04:37.869967 24368 catalog_manager.cc:5495] T e6b6ec010caf4689b8c90449764f5cf9 P 1bc1f2e62ab44af995165bd5116a9d43 reported cstate change: term changed from 2 to 3, leader changed from 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1) to 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4). New cstate: current_term: 3 leader_uuid: "1bc1f2e62ab44af995165bd5116a9d43" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260925 03:04:38.279495 24569 debug-util.cc:398] Leaking SignalData structure 0x7b0800201ae0 after lost signal to thread 23303
I20260925 03:04:38.279338 26209 raft_consensus.cc:504] T a7aa47c48404465b94a36fd819fe3c7a P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:38.279455 26240 raft_consensus.cc:504] T a4abe0ea757e4886be4945c5001e529a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
W20260925 03:04:38.280709 24569 debug-util.cc:398] Leaking SignalData structure 0x7b0800118e40 after lost signal to thread 24405
W20260925 03:04:38.281088 24569 debug-util.cc:398] Leaking SignalData structure 0x7b08001f2340 after lost signal to thread 24418
I20260925 03:04:38.280433 26083 raft_consensus.cc:504] T ac9357d144aa49789faaf31fe5e6bf89 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:38.281828 26240 leader_election.cc:288] T a4abe0ea757e4886be4945c5001e529a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:38.281999 26209 leader_election.cc:288] T a7aa47c48404465b94a36fd819fe3c7a P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
W20260925 03:04:38.282284 24569 debug-util.cc:398] Leaking SignalData structure 0x7b080038ab00 after lost signal to thread 24497
W20260925 03:04:38.128041 24468 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1176) took 3103 ms (client timeout 2103 ms). Trace:
I20260925 03:04:38.282478 24469 raft_consensus.cc:2362] T a7aa47c48404465b94a36fd819fe3c7a P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 2. Re-sending same reply.
W20260925 03:04:38.282649 24468 rpcz_store.cc:269] 0925 03:04:35.024237 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:38.127749 (+3103512us) service_pool.cc:214] Skipping call since client already timed out
0925 03:04:38.127940 (+ 191us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20260925 03:04:38.283517 24468 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1193) took 2471 ms (client timeout 1505 ms). Trace:
W20260925 03:04:38.283833 24468 rpcz_store.cc:269] 0925 03:04:35.811980 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:38.283354 (+2471374us) service_pool.cc:214] Skipping call since client already timed out
0925 03:04:38.283461 (+ 107us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20260925 03:04:38.130374 25163 leader_election.cc:334] T 29637f5a4a504ce6ba29033ccd0cd716 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389): Timed out: RequestConsensusVote RPC to 127.22.193.2:36389 timed out after 1.969s (SENT)
I20260925 03:04:38.284905 25163 leader_election.cc:302] T 29637f5a4a504ce6ba29033ccd0cd716 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
W20260925 03:04:38.285375 24569 debug-util.cc:398] Leaking SignalData structure 0x7b080038ab80 after lost signal to thread 24498
W20260925 03:04:38.145881 25220 debug-util.cc:398] Leaking SignalData structure 0x7b08002dc8a0 after lost signal to thread 23303
W20260925 03:04:38.286180 24569 debug-util.cc:398] Leaking SignalData structure 0x7b0800394f00 after lost signal to thread 24544
I20260925 03:04:38.284998 26083 leader_election.cc:288] T ac9357d144aa49789faaf31fe5e6bf89 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
W20260925 03:04:38.286494 24569 debug-util.cc:398] Leaking SignalData structure 0x7b0800394de0 after lost signal to thread 24547
W20260925 03:04:38.286695 25220 debug-util.cc:398] Leaking SignalData structure 0x7b08002dcf80 after lost signal to thread 24405
W20260925 03:04:38.287053 25220 debug-util.cc:398] Leaking SignalData structure 0x7b08002dd040 after lost signal to thread 24418
W20260925 03:04:38.287094 24569 debug-util.cc:398] Leaking SignalData structure 0x7b0800394f80 after lost signal to thread 24572
W20260925 03:04:38.288295 25220 debug-util.cc:398] Leaking SignalData structure 0x7b08000b8640 after lost signal to thread 24497
W20260925 03:04:38.149351 24436 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 2.502s user 0.036s sys 0.062s
I20260925 03:04:38.156535 24439 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 776) took 2419 ms. Trace:
I20260925 03:04:38.288915 24439 rpcz_store.cc:276] 0925 03:04:35.736877 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:38.156486 (+2419609us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260925 03:04:38.290009 26077 raft_consensus.cc:2726] T 29637f5a4a504ce6ba29033ccd0cd716 P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260925 03:04:38.155767 24437 service_pool.cc:137] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:55562 dropped due to backpressure. The service queue is full; it has 50 items. [suppressed 6 similar messages]
I20260925 03:04:38.290870 24437 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 777) took 2553 ms. Trace:
I20260925 03:04:38.291188 24437 rpcz_store.cc:276] 0925 03:04:35.737217 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:38.290804 (+2553587us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260925 03:04:38.289611 24439 service_pool.cc:140] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:55562 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1203)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1199)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 750)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 764)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 780)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1196)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 771)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1197)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1200)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 766)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1201)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1195)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1204)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1192)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1194)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1202)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 770)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 762)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1205)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1206)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 781)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 741)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 742)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 745)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 746)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 747)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55560 (request call id 1172)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55560 (request call id 1178)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55560 (request call id 1180)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55560 (request call id 1183)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55560 (request call id 1184)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55560 (request call id 1185)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55560 (request call id 1191)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 756)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 759)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 761)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 763)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 765)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 767)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 768)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 774)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 775)
W20260925 03:04:38.179658 24465 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1175) took 3715 ms (client timeout 2016 ms). Trace:
I20260925 03:04:38.291872 24511 leader_election.cc:302] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
W20260925 03:04:38.292119 24465 rpcz_store.cc:269] 0925 03:04:34.464040 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:35.661139 (+1197099us) service_pool.cc:227] Handling call
0925 03:04:38.179362 (+2518223us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260925 03:04:38.186103 24467 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 757) took 2739 ms (client timeout 1869 ms). Trace:
W20260925 03:04:38.293021 24467 rpcz_store.cc:269] 0925 03:04:35.447115 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:38.185950 (+2738835us) service_pool.cc:214] Skipping call since client already timed out
0925 03:04:38.186048 (+ 98us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20260925 03:04:38.186554 24470 rpcz_store.cc:269] 0925 03:04:31.492514 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:31.492707 (+ 193us) service_pool.cc:227] Handling call
0925 03:04:31.968401 (+475694us) spinlock_profiling.cc:243] Waited 4.14 ms on lock 0x7b5800001598. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f43733ecf11 00007f4373570851 00007f4373570702 00007f437c460b09 00007f437b39c730 00007f437b46279b 00007f437b462821 00007f437b4121b0 00007f437b411a94 00007f4374af9c04 00007f4374af9b92
0925 03:04:35.507642 (+3539241us) spinlock_profiling.cc:243] Waited 2.93 s on lock 0x7b58002e31b0. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f437644bf63 00007f43764afaba 00007f43764c73af 00007f43764c8b4e 00007f43764d4228 00007f437b411c27 00007f4374af9c04 00007f4374af9b92 00007f4374af9ac1 00007f4374af9a3c
0925 03:04:35.816250 (+308608us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6737111040}
W20260925 03:04:38.191932 24471 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1179) took 3166 ms (client timeout 2083 ms). Trace:
I20260925 03:04:38.291662 24437 service_pool.cc:140] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:55562 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1203)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1199)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 750)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 764)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 780)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1196)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 771)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1197)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1200)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 766)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1201)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1195)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1204)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1192)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1194)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1202)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 770)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 762)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1205)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1206)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 781)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 741)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 742)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 745)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 746)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 747)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55560 (request call id 1172)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55560 (request call id 1178)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55560 (request call id 1180)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55560 (request call id 1183)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55560 (request call id 1184)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55560 (request call id 1185)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55560 (request call id 1191)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 756)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 759)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 761)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 763)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 765)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 767)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 768)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 774)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 775)
W20260925 03:04:38.293990 24471 rpcz_store.cc:269] 0925 03:04:35.025429 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:35.837092 (+811663us) service_pool.cc:227] Handling call
0925 03:04:38.191870 (+2354778us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260925 03:04:38.184564 24466 rpcz_store.cc:269] 0925 03:04:34.339386 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:38.184298 (+3844912us) service_pool.cc:214] Skipping call since client already timed out
0925 03:04:38.184360 (+ 62us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20260925 03:04:38.294764 24471 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1203) took 2470 ms (client timeout 1728 ms). Trace:
W20260925 03:04:38.192816 24472 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 769) took 2742 ms (client timeout 1626 ms). Trace:
W20260925 03:04:38.193073 24473 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 755) took 3730 ms (client timeout 2028 ms). Trace:
W20260925 03:04:38.295475 24472 rpcz_store.cc:269] 0925 03:04:35.450438 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:35.804370 (+353932us) service_pool.cc:227] Handling call
0925 03:04:38.192742 (+2388372us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:38.199716 24540 raft_consensus.cc:2454] T 21c43af5e6844fdaacc8c625fa01fd8d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 2.
W20260925 03:04:38.296197 24540 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 1266) took 2485 ms (client timeout 1717 ms). Trace:
W20260925 03:04:38.201357 24543 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36786 (request call id 710) took 2816 ms (client timeout 2185 ms). Trace:
I20260925 03:04:35.822153 24544 raft_consensus.cc:2454] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 3.
W20260925 03:04:38.296916 24436 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 2.650s user 0.036s sys 0.062s
W20260925 03:04:38.296990 24544 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36786 (request call id 726) took 2670 ms (client timeout 1729 ms). Trace:
W20260925 03:04:38.201685 24541 rpcz_store.cc:269] 0925 03:04:33.874173 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:35.627718 (+1753545us) service_pool.cc:227] Handling call
0925 03:04:38.200962 (+2573244us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260925 03:04:38.203346 24539 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 1264) took 2504 ms (client timeout 1873 ms). Trace:
I20260925 03:04:38.292896 24511 leader_election.cc:302] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters: 1bc1f2e62ab44af995165bd5116a9d43
W20260925 03:04:38.297569 24539 rpcz_store.cc:269] 0925 03:04:35.698926 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:38.203122 (+2504196us) service_pool.cc:214] Skipping call since client already timed out
0925 03:04:38.203264 (+ 142us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260925 03:04:38.298445 24511 leader_election.cc:302] T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:38.298552 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:38.299242 24539 raft_consensus.cc:2454] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 4.
I20260925 03:04:35.826078 24547 raft_consensus.cc:2379] T 8ac466793d094325ace224708b2158ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c in current term 3: Already voted for candidate 1bc1f2e62ab44af995165bd5116a9d43 in this term.
I20260925 03:04:38.299485 24511 leader_election.cc:302] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
W20260925 03:04:38.299923 24547 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36786 (request call id 705) took 2916 ms (client timeout 2080 ms). Trace:
I20260925 03:04:38.234894 26115 raft_consensus.cc:482] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:38.236727 24615 rpcz_store.cc:276] 0925 03:04:34.531257 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:35.695845 (+1164588us) service_pool.cc:227] Handling call
0925 03:04:35.696208 (+ 363us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:35.799155 (+102947us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:35.799188 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:35.799225 (+ 37us) raft_consensus.cc:1631] Marking committed up to 5
0925 03:04:35.799254 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:35.799316 (+ 62us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:35.799439 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":170,"raft.queue_time_us":138,"raft.run_cpu_time_us":154,"raft.run_wall_time_us":5938}
I20260925 03:04:38.300233 25804 raft_consensus.cc:2781] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260925 03:04:38.240454 24619 rpcz_store.cc:276] 0925 03:04:34.528759 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:35.677966 (+1149207us) service_pool.cc:227] Handling call
0925 03:04:35.806073 (+128107us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:38.300514 26115 raft_consensus.cc:3037] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:38.300925 25804 raft_consensus.cc:482] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 1bc1f2e62ab44af995165bd5116a9d43)
I20260925 03:04:38.301326 25804 raft_consensus.cc:3037] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:38.302168 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35992316cb8d4c028333388eff02ecde" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
W20260925 03:04:38.302963 24541 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36786 (request call id 715) took 2915 ms (client timeout 3199 ms). Trace:
W20260925 03:04:38.303318 24541 rpcz_store.cc:269] 0925 03:04:35.387819 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:38.301863 (+2914044us) service_pool.cc:227] Handling call
0925 03:04:38.302912 (+ 1049us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:38.304229 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5278a60f97da4e10aecd50a8d9a9a43c" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:38.300042 26138 raft_consensus.cc:2781] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260925 03:04:38.304885 26138 raft_consensus.cc:482] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:38.305262 26138 raft_consensus.cc:3037] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Advancing to term 5
W20260925 03:04:38.307538 24470 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 750) took 3888 ms (client timeout 3166 ms). Trace:
I20260925 03:04:38.300513 26107 raft_consensus.cc:2781] T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260925 03:04:38.295841 24473 rpcz_store.cc:269] 0925 03:04:34.463153 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:35.671325 (+1208172us) service_pool.cc:227] Handling call
0925 03:04:38.193017 (+2521692us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:38.307938 25804 raft_consensus.cc:504] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:38.308769 26107 raft_consensus.cc:482] T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1bc1f2e62ab44af995165bd5116a9d43)
W20260925 03:04:38.309226 24473 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 764) took 2860 ms (client timeout 2185 ms). Trace:
W20260925 03:04:38.309569 24473 rpcz_store.cc:269] 0925 03:04:35.449284 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:38.309065 (+2859781us) service_pool.cc:214] Skipping call since client already timed out
0925 03:04:38.309170 (+ 105us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20260925 03:04:38.296535 24540 rpcz_store.cc:269] 0925 03:04:35.811062 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:35.852370 (+ 41308us) service_pool.cc:227] Handling call
0925 03:04:38.296132 (+2443762us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260925 03:04:38.311170 24436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 2.65013e+06us
W20260925 03:04:38.311931 24467 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 771) took 2861 ms (client timeout 2242 ms). Trace:
W20260925 03:04:38.300277 24547 rpcz_store.cc:269] 0925 03:04:35.383397 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:35.708125 (+324728us) service_pool.cc:227] Handling call
0925 03:04:38.299859 (+2591734us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:38.313678 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc2df40db08649c9a75e90000b878559" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
W20260925 03:04:38.305097 24543 rpcz_store.cc:269] 0925 03:04:35.386300 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:35.880165 (+493865us) service_pool.cc:227] Handling call
0925 03:04:38.201294 (+2321129us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:38.310945 26196 raft_consensus.cc:3037] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Advancing to term 5
W20260925 03:04:38.295329 24466 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1199) took 2481 ms (client timeout 1770 ms). Trace:
I20260925 03:04:38.310460 25804 leader_election.cc:288] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 election: Requested vote from peers eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:38.316774 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7993f9c3e2a44f408bbf5d1e395e2306" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:38.317466 24540 raft_consensus.cc:2454] T 7993f9c3e2a44f408bbf5d1e395e2306 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 3.
I20260925 03:04:38.316188 25804 raft_consensus.cc:2781] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260925 03:04:38.315079 24544 rpcz_store.cc:269] 0925 03:04:35.626523 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:35.692779 (+ 66256us) service_pool.cc:227] Handling call
0925 03:04:38.296912 (+2604133us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:38.319419 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be4e0aa984f4f2b81a58e382e434ad5" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:38.320055 24544 raft_consensus.cc:2454] T 0be4e0aa984f4f2b81a58e382e434ad5 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 3.
I20260925 03:04:38.318454 25804 raft_consensus.cc:482] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:38.320730 25804 raft_consensus.cc:3037] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:38.315125 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9c33fd100ff44c1a3247319a324a044" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
W20260925 03:04:38.315815 24466 rpcz_store.cc:269] 0925 03:04:35.813667 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:38.295173 (+2481506us) service_pool.cc:214] Skipping call since client already timed out
0925 03:04:38.295273 (+ 100us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20260925 03:04:38.284534 24468 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 758) took 2837 ms (client timeout 2095 ms). Trace:
W20260925 03:04:38.310344 24472 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 780) took 2572 ms (client timeout 1929 ms). Trace:
W20260925 03:04:38.321745 24468 rpcz_store.cc:269] 0925 03:04:35.447437 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:38.284388 (+2836951us) service_pool.cc:214] Skipping call since client already timed out
0925 03:04:38.284481 (+ 93us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20260925 03:04:38.285563 25163 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 155231us
W20260925 03:04:38.295120 24471 rpcz_store.cc:269] 0925 03:04:35.824653 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:38.294603 (+2469950us) service_pool.cc:214] Skipping call since client already timed out
0925 03:04:38.294705 (+ 102us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20260925 03:04:38.322840 25163 leader_election.cc:334] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389): Timed out: RequestConsensusVote RPC to 127.22.193.2:36389 timed out after 2.052s (SENT)
W20260925 03:04:38.323143 24471 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 766) took 2873 ms (client timeout 2281 ms). Trace:
I20260925 03:04:38.250725 25979 raft_consensus.cc:482] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260925 03:04:38.323499 24471 rpcz_store.cc:269] 0925 03:04:35.449782 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:38.322993 (+2873211us) service_pool.cc:214] Skipping call since client already timed out
0925 03:04:38.323090 (+ 97us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260925 03:04:38.250838 25942 raft_consensus.cc:2781] T a7aa47c48404465b94a36fd819fe3c7a P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Leader election won for term 2
I20260925 03:04:38.255533 26068 leader_election.cc:288] T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:38.255192 26076 raft_consensus.cc:504] T 7e49248643df4131a1c42e74244e05c2 P 1bc1f2e62ab44af995165bd5116a9d43 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:35.826906 26086 raft_consensus.cc:482] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260925 03:04:38.324255 24588 connection.cc:232] RPC call timeout handler was delayed by -2.2427097010004218s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1869s for a response
W20260925 03:04:38.321627 24466 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1200) took 2507 ms (client timeout 1912 ms). Trace:
I20260925 03:04:38.241104 25194 rpcz_store.cc:276] 0925 03:04:35.726414 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:35.729573 (+ 3159us) service_pool.cc:227] Handling call
0925 03:04:35.729947 (+ 374us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:35.730165 (+ 218us) raft_consensus.cc:1520] Early marking committed up to index 4
0925 03:04:36.153927 (+423762us) log.cc:842] Serialized 14 byte log entry
0925 03:04:36.754297 (+600370us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:36.754975 (+ 678us) raft_consensus.cc:1631] Marking committed up to 4
0925 03:04:36.755519 (+ 544us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:36.756304 (+ 785us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:36.757868 (+ 1564us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":84950,"prepare.run_cpu_time_us":55,"prepare.run_wall_time_us":56,"raft.queue_time_us":66296,"raft.run_cpu_time_us":117,"raft.run_wall_time_us":822,"thread_start_us":801707,"threads_started":3,"wal-append.queue_time_us":572072,"wal-append.run_cpu_time_us":1291,"wal-append.run_wall_time_us":1002012}
I20260925 03:04:38.269349 26120 raft_consensus.cc:3037] T 74443a0a1dc34d88a2f42e3b722db405 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:38.270004 26241 raft_consensus.cc:482] T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:38.273350 26175 raft_consensus.cc:3037] T 37a157996e944db2906110d87ecb0c54 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:38.274502 24621 raft_consensus.cc:2354] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 3. Current term is 4.
I20260925 03:04:38.275439 26181 leader_election.cc:288] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:38.272666 26150 leader_election.cc:288] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:38.278162 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b627ecb4a5384ef392aa695e06599c39" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:38.285004 26084 raft_consensus.cc:3037] T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:38.296555 24464 raft_consensus.cc:2354] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 4. Current term is 5.
I20260925 03:04:38.224561 26239 raft_consensus.cc:504] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:38.210273 24548 raft_consensus.cc:2454] T edcf6e6a422d4d198d4fbb0f7d3d845d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 1.
I20260925 03:04:38.242568 25982 raft_consensus.cc:504] T b1e85248d3b04b979dd080caeeb02fdb P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:38.309343 26107 raft_consensus.cc:3037] T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Advancing to term 3
W20260925 03:04:38.323763 24585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.4:43979, user_credentials={real_user=slave}} blocked reactor thread for 2.41365e+06us
W20260925 03:04:38.310402 24473 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1196) took 2497 ms (client timeout 1879 ms). Trace:
W20260925 03:04:38.264880 24546 rpcz_store.cc:269] 0925 03:04:35.834036 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:38.262935 (+2428899us) service_pool.cc:227] Handling call
0925 03:04:38.264194 (+ 1259us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260925 03:04:38.324219 24471 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1201) took 2505 ms (client timeout 1916 ms). Trace:
I20260925 03:04:38.355589 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6b6ec010caf4689b8c90449764f5cf9" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
W20260925 03:04:38.312352 24467 rpcz_store.cc:269] 0925 03:04:35.450878 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:38.311752 (+2860874us) service_pool.cc:214] Skipping call since client already timed out
0925 03:04:38.311869 (+ 117us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260925 03:04:38.314215 24436 leader_election.cc:302] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
W20260925 03:04:38.322247 24472 rpcz_store.cc:269] 0925 03:04:35.738058 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:38.310168 (+2572110us) service_pool.cc:214] Skipping call since client already timed out
0925 03:04:38.310281 (+ 113us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20260925 03:04:38.356552 24468 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1195) took 2544 ms (client timeout 1936 ms). Trace:
I20260925 03:04:38.318367 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442d4a29b1d743599f4b6b4871e26668" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
W20260925 03:04:38.307919 24470 rpcz_store.cc:269] 0925 03:04:34.419014 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:38.307358 (+3888344us) service_pool.cc:214] Skipping call since client already timed out
0925 03:04:38.307476 (+ 118us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260925 03:04:38.314285 24547 raft_consensus.cc:2454] T cc2df40db08649c9a75e90000b878559 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 3.
W20260925 03:04:38.330586 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 2.51008e+06us
W20260925 03:04:38.345112 24511 connection.cc:232] RPC call timeout handler was delayed by -2.7349218700001074s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1912s for a response
W20260925 03:04:38.284531 24469 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 772) took 2833 ms (client timeout 1630 ms). Trace:
I20260925 03:04:38.333467 25942 raft_consensus.cc:686] T a7aa47c48404465b94a36fd819fe3c7a P eeab93abfdb141a895002ec9f312296c [term 2 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
W20260925 03:04:38.333881 24466 rpcz_store.cc:269] 0925 03:04:35.813906 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:38.321462 (+2507556us) service_pool.cc:214] Skipping call since client already timed out
0925 03:04:38.321575 (+ 113us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260925 03:04:38.453711 25198 raft_consensus.cc:2354] T 7e49248643df4131a1c42e74244e05c2 P 1bc1f2e62ab44af995165bd5116a9d43 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c for earlier term 4. Current term is 5.
W20260925 03:04:38.456159 24466 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1204) took 2628 ms (client timeout 2029 ms). Trace:
I20260925 03:04:38.457154 24545 raft_consensus.cc:2454] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 4.
W20260925 03:04:38.458155 24545 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 1261) took 2852 ms (client timeout 2709 ms). Trace:
W20260925 03:04:38.458494 24545 rpcz_store.cc:269] 0925 03:04:35.606033 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:38.220063 (+2614030us) service_pool.cc:227] Handling call
0925 03:04:38.457987 (+237924us) spinlock_profiling.cc:243] Waited 200 ms on lock 0x7b54002828d4. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d42ae 00007f437b411c27 00007f4374af9c04 00007f4374af9b92 00007f4374af9ac1 00007f4374af9a3c 00007f4374af8cf2 00007f43748fa94c 00007f43748f9d86 00007f43748f972f
0925 03:04:38.458091 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":459838336}
I20260925 03:04:38.459873 24545 raft_consensus.cc:1261] T a4abe0ea757e4886be4945c5001e529a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260925 03:04:38.460399 24545 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 680) took 5556 ms. Trace:
W20260925 03:04:38.461266 25198 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49794 (request call id 159) took 2832 ms (client timeout 1857 ms). Trace:
W20260925 03:04:38.316051 24465 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1197) took 2503 ms (client timeout 1884 ms). Trace:
W20260925 03:04:38.463578 25165 leader_election.cc:334] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187): Timed out: RequestConsensusVote RPC to 127.22.193.1:42187 timed out after 2.052s (SENT)
I20260925 03:04:38.460682 24545 rpcz_store.cc:276] 0925 03:04:32.903779 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:38.459247 (+5555468us) service_pool.cc:227] Handling call
0925 03:04:38.459553 (+ 306us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:38.460169 (+ 616us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:38.460340 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:38.464015 25165 leader_election.cc:302] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
W20260925 03:04:38.465343 25165 leader_election.cc:334] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187): Timed out: RequestConsensusVote RPC to 127.22.193.1:42187 timed out after 2.126s (SENT)
I20260925 03:04:38.465518 24545 raft_consensus.cc:1261] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260925 03:04:38.465803 25165 leader_election.cc:302] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
W20260925 03:04:38.365747 24621 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38564 (request call id 1220) took 3870 ms (client timeout 1728 ms). Trace:
I20260925 03:04:38.363364 24542 raft_consensus.cc:2454] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 4.
W20260925 03:04:38.466521 25198 rpcz_store.cc:269] 0925 03:04:35.633015 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:35.640865 (+ 7850us) service_pool.cc:227] Handling call
0925 03:04:38.461084 (+2820219us) spinlock_profiling.cc:243] Waited 2.8 s on lock 0x7b54003c7dd4. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d42ae 00007f437b411c27 00007f4374af9c04 00007f4374af9b92 00007f4374af9ac1 00007f4374af9a3c 00007f4374af8cf2 00007f43748fa94c 00007f43748f9d86 00007f43748f972f
0925 03:04:38.461208 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6451484416}
W20260925 03:04:38.468299 24542 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 1265) took 2769 ms (client timeout 2807 ms). Trace:
I20260925 03:04:38.364392 25199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71f461a6e2ff434993b78a2d077f0abd" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43"
I20260925 03:04:38.469754 24540 raft_consensus.cc:3037] T 442d4a29b1d743599f4b6b4871e26668 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Advancing to term 5
W20260925 03:04:38.288316 24569 debug-util.cc:398] Leaking SignalData structure 0x7b0800320c20 after lost signal to thread 24646
I20260925 03:04:38.466325 24545 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 682) took 5557 ms. Trace:
I20260925 03:04:38.471388 24545 rpcz_store.cc:276] 0925 03:04:32.909419 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:38.464911 (+5555492us) service_pool.cc:227] Handling call
0925 03:04:38.465204 (+ 293us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:38.466051 (+ 847us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:38.466277 (+ 226us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260925 03:04:38.473577 24548 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36786 (request call id 722) took 2886 ms (client timeout 1929 ms). Trace:
W20260925 03:04:38.469480 24542 rpcz_store.cc:269] 0925 03:04:35.767341 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:38.277885 (+2510544us) service_pool.cc:227] Handling call
0925 03:04:38.468237 (+190352us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260925 03:04:38.467666 24621 rpcz_store.cc:269] 0925 03:04:34.495617 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:34.518016 (+ 22399us) service_pool.cc:227] Handling call
0925 03:04:38.365518 (+3847502us) spinlock_profiling.cc:243] Waited 2.72 s on lock 0x7b54003241d4. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d42ae 00007f437b411c27 00007f4374af9c04 00007f4374af9b92 00007f4374af9ac1 00007f4374af9a3c 00007f4374af8cf2 00007f43748fa94c 00007f43748f9d86 00007f43748f972f
0925 03:04:38.365679 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6265420544}
I20260925 03:04:38.474365 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9c33fd100ff44c1a3247319a324a044" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:38.473387 25198 raft_consensus.cc:1261] T 367277daf8d04778b3fa4ed94d71580d P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 LEARNER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
W20260925 03:04:38.474903 24464 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1181) took 3448 ms (client timeout 1858 ms). Trace:
W20260925 03:04:38.475353 24464 rpcz_store.cc:269] 0925 03:04:35.026000 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:35.799933 (+773933us) service_pool.cc:227] Handling call
0925 03:04:38.474533 (+2674600us) spinlock_profiling.cc:243] Waited 139 ms on lock 0x7b54001923d4. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d42ae 00007f437b411c27 00007f4374af9c04 00007f4374af9b92 00007f4374af9ac1 00007f4374af9a3c 00007f4374af8cf2 00007f43748fa94c 00007f43748f9d86 00007f43748f972f
0925 03:04:38.474840 (+ 307us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":319840512}
W20260925 03:04:38.475860 24472 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1194) took 2663 ms (client timeout 2160 ms). Trace:
W20260925 03:04:38.456561 24466 rpcz_store.cc:269] 0925 03:04:35.827714 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:38.455981 (+2628267us) service_pool.cc:214] Skipping call since client already timed out
0925 03:04:38.456103 (+ 122us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20260925 03:04:38.476235 24472 rpcz_store.cc:269] 0925 03:04:35.812272 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:38.475718 (+2663446us) service_pool.cc:214] Skipping call since client already timed out
0925 03:04:38.475810 (+ 92us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20260925 03:04:38.477501 24468 rpcz_store.cc:269] 0925 03:04:35.812545 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:38.356382 (+2543837us) service_pool.cc:214] Skipping call since client already timed out
0925 03:04:38.356496 (+ 114us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20260925 03:04:38.478032 24586 connection.cc:232] RPC call timeout handler was delayed by -2.4705280979997042s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1626s for a response
W20260925 03:04:38.478761 24586 connection.cc:232] RPC call timeout handler was delayed by -2.1087280980009382s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2028s for a response
I20260925 03:04:38.333861 26086 raft_consensus.cc:504] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
W20260925 03:04:38.479122 24586 connection.cc:232] RPC call timeout handler was delayed by -2.0619280980008625s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20800000000000002s for a response
W20260925 03:04:38.507512 24586 connection.cc:232] RPC call timeout handler was delayed by -2.0484280979999312s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20950000000000002s for a response
W20260925 03:04:38.507927 24586 connection.cc:232] RPC call timeout handler was delayed by -2.0359240890011279s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3166s for a response
W20260925 03:04:38.508244 24586 connection.cc:232] RPC call timeout handler was delayed by -1.9674280979998002s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21850000000000003s for a response
W20260925 03:04:38.469393 24473 rpcz_store.cc:269] 0925 03:04:35.812852 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:38.310257 (+2497405us) service_pool.cc:214] Skipping call since client already timed out
0925 03:04:38.310350 (+ 93us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20260925 03:04:38.478235 24471 rpcz_store.cc:269] 0925 03:04:35.819349 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:38.324086 (+2504737us) service_pool.cc:214] Skipping call since client already timed out
0925 03:04:38.324175 (+ 89us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260925 03:04:38.467550 24544 raft_consensus.cc:2454] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 3.
W20260925 03:04:38.509457 24511 connection.cc:232] RPC call timeout handler was delayed by -2.4936609859978489s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1824s for a response
W20260925 03:04:38.509950 24511 connection.cc:232] RPC call timeout handler was delayed by -1.5248454069987929s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1626s for a response
W20260925 03:04:38.467777 24547 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36786 (request call id 724) took 2874 ms (client timeout 3095 ms). Trace:
W20260925 03:04:38.510407 24511 connection.cc:232] RPC call timeout handler was delayed by -1.1297454069990636s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20650000000000002s for a response
I20260925 03:04:38.510500 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b8e0d260a884ecdb20f79de46c36b19" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
W20260925 03:04:38.510751 24511 connection.cc:232] RPC call timeout handler was delayed by -1.1036881759991957s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3179s for a response
W20260925 03:04:38.510416 24547 rpcz_store.cc:269] 0925 03:04:35.592958 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:38.313416 (+2720458us) service_pool.cc:227] Handling call
0925 03:04:38.467727 (+154311us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:38.478904 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5278a60f97da4e10aecd50a8d9a9a43c" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
W20260925 03:04:38.360930 24539 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 1263) took 2659 ms (client timeout 2875 ms). Trace:
I20260925 03:04:38.366849 25195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4abe0ea757e4886be4945c5001e529a" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:38.366518 25193 raft_consensus.cc:1261] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 FOLLOWER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260925 03:04:38.505007 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5aec38e66a947c681e6c2a72b05e6ca" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
W20260925 03:04:38.464030 24465 rpcz_store.cc:269] 0925 03:04:35.813118 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:38.315863 (+2502745us) service_pool.cc:214] Skipping call since client already timed out
0925 03:04:38.315989 (+ 126us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260925 03:04:38.505239 25942 consensus_queue.cc:237] T a7aa47c48404465b94a36fd819fe3c7a P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:38.477293 26209 raft_consensus.cc:482] T 9e8af2641e124f9589c5ccfdac785f63 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:38.507776 24545 raft_consensus.cc:1261] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260925 03:04:38.511410 24542 raft_consensus.cc:3037] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:38.511677 24544 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 686) took 5600 ms. Trace:
W20260925 03:04:38.469069 24467 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1192) took 2657 ms (client timeout 2110 ms). Trace:
I20260925 03:04:38.472084 26077 raft_consensus.cc:2726] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260925 03:04:38.512509 24470 raft_consensus.cc:2454] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 2.
W20260925 03:04:38.525492 24470 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 762) took 3076 ms (client timeout 3273 ms). Trace:
I20260925 03:04:38.476401 26150 raft_consensus.cc:2726] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260925 03:04:38.502656 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60d45b7d467d4e1ab92367f6d7d52b15" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:38.364737 25196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ac466793d094325ace224708b2158ed" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
W20260925 03:04:38.516510 24586 connection.cc:232] RPC call timeout handler was delayed by -1.8810280979996605s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22810000000000002s for a response
I20260925 03:04:38.525828 24540 raft_consensus.cc:2454] T 442d4a29b1d743599f4b6b4871e26668 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 5.
I20260925 03:04:38.518402 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e8af2641e124f9589c5ccfdac785f63" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
W20260925 03:04:38.473979 24548 rpcz_store.cc:269] 0925 03:04:35.587622 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:35.805672 (+218050us) service_pool.cc:227] Handling call
0925 03:04:38.473514 (+2667842us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:38.477279 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35992316cb8d4c028333388eff02ecde" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:38.528093 24466 raft_consensus.cc:2454] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 4.
W20260925 03:04:38.522466 24467 rpcz_store.cc:269] 0925 03:04:35.811679 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:38.468908 (+2657229us) service_pool.cc:214] Skipping call since client already timed out
0925 03:04:38.469017 (+ 109us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260925 03:04:38.332000 24541 raft_consensus.cc:2454] T 5278a60f97da4e10aecd50a8d9a9a43c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 2.
I20260925 03:04:38.523015 24545 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 684) took 5612 ms. Trace:
I20260925 03:04:38.479789 25938 raft_consensus.cc:482] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:38.362867 26241 raft_consensus.cc:504] T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:38.523711 26209 raft_consensus.cc:504] T 9e8af2641e124f9589c5ccfdac785f63 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:38.361533 25979 raft_consensus.cc:504] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
W20260925 03:04:38.526860 24586 connection.cc:232] RPC call timeout handler was delayed by -1.73252809800033s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.24460000000000004s for a response
W20260925 03:04:38.539227 24586 connection.cc:232] RPC call timeout handler was delayed by -1.2833880050002335s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17290000000000003s for a response
W20260925 03:04:38.539575 24586 connection.cc:232] RPC call timeout handler was delayed by -1.1428388450003695s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19290000000000002s for a response
W20260925 03:04:38.539837 24586 connection.cc:232] RPC call timeout handler was delayed by -1.0548280980001437s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3199s for a response
W20260925 03:04:38.482584 25220 debug-util.cc:398] Leaking SignalData structure 0x7b080038ad00 after lost signal to thread 24572
I20260925 03:04:38.524887 26246 raft_consensus.cc:482] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:38.541508 26246 raft_consensus.cc:504] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:04:38.542243 25220 debug-util.cc:398] Leaking SignalData structure 0x7b08002c6540 after lost signal to thread 24646
W20260925 03:04:38.543515 25220 debug-util.cc:398] Leaking SignalData structure 0x7b080033a1e0 after lost signal to thread 25223
W20260925 03:04:38.455771 24469 rpcz_store.cc:269] 0925 03:04:35.451135 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:35.819837 (+368702us) service_pool.cc:227] Handling call
0925 03:04:38.284331 (+2464494us) spinlock_profiling.cc:243] Waited 2.46 s on lock 0x7b54003eb7d4. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d42ae 00007f437b411c27 00007f4374af9c04 00007f4374af9b92 00007f4374af9ac1 00007f4374af9a3c 00007f4374af8cf2 00007f43748fa94c 00007f43748f9d86 00007f43748f972f
0925 03:04:38.284453 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5650286336}
I20260925 03:04:38.519531 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9357d144aa49789faaf31fe5e6bf89" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:38.509326 25897 raft_consensus.cc:482] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:38.529392 24545 rpcz_store.cc:276] 0925 03:04:32.910330 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:38.507127 (+5596797us) service_pool.cc:227] Handling call
0925 03:04:38.507473 (+ 346us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:38.522759 (+ 15286us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:38.522950 (+ 191us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:38.556344 24545 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 695) took 4929 ms. Trace:
I20260925 03:04:38.556864 24545 rpcz_store.cc:276] 0925 03:04:33.626884 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:38.555601 (+4928717us) service_pool.cc:227] Handling call
0925 03:04:38.555895 (+ 294us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:38.556054 (+ 159us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:38.556077 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:38.556115 (+ 38us) raft_consensus.cc:1631] Marking committed up to 3
0925 03:04:38.556140 (+ 25us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:38.556188 (+ 48us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:38.556289 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:38.559016 24545 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 1243) took 4924 ms. Trace:
I20260925 03:04:38.515110 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be4e0aa984f4f2b81a58e382e434ad5" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:38.524466 25195 raft_consensus.cc:2396] T a4abe0ea757e4886be4945c5001e529a P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for term 3 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260925 03:04:38.567808 24464 raft_consensus.cc:2396] T 0be4e0aa984f4f2b81a58e382e434ad5 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260925 03:04:38.532672 24540 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 1241) took 4913 ms. Trace:
I20260925 03:04:38.536653 24542 raft_consensus.cc:2454] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 4.
I20260925 03:04:38.513159 26244 raft_consensus.cc:482] T 367277daf8d04778b3fa4ed94d71580d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:38.522222 24544 rpcz_store.cc:276] 0925 03:04:32.911346 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:38.509784 (+5598438us) service_pool.cc:227] Handling call
0925 03:04:38.510055 (+ 271us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:38.510449 (+ 394us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:38.510489 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:38.510533 (+ 44us) raft_consensus.cc:1631] Marking committed up to 3
0925 03:04:38.510566 (+ 33us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:38.510620 (+ 54us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:38.511626 (+ 1006us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260925 03:04:38.524971 24569 debug-util.cc:398] Leaking SignalData structure 0x7b0800341520 after lost signal to thread 25223
I20260925 03:04:38.544231 24468 raft_consensus.cc:2454] T 9e8af2641e124f9589c5ccfdac785f63 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 3.
I20260925 03:04:38.544426 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "293ef69caf514ac98510a68495f54eda" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:38.571873 24544 raft_consensus.cc:1261] T cc2df40db08649c9a75e90000b878559 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260925 03:04:38.573274 24544 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 1247) took 4858 ms. Trace:
I20260925 03:04:38.573611 24544 rpcz_store.cc:276] 0925 03:04:33.716616 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:38.571298 (+4854682us) service_pool.cc:227] Handling call
0925 03:04:38.571587 (+ 289us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:38.573014 (+ 1427us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:38.573219 (+ 205us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:38.574597 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e8af2641e124f9589c5ccfdac785f63" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:38.575769 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c23337a7cb74184a0e57b7095cb6a11" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
W20260925 03:04:38.527127 24470 rpcz_store.cc:269] 0925 03:04:35.448656 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:38.478686 (+3030030us) service_pool.cc:227] Handling call
0925 03:04:38.525424 (+ 46738us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:38.546309 24548 raft_consensus.cc:1261] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
I20260925 03:04:38.576939 24548 pending_rounds.cc:76] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5: Aborting all ops after (but not including) 1
I20260925 03:04:38.577178 24548 pending_rounds.cc:97] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20260925 03:04:38.577416 24548 raft_consensus.cc:2866] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260925 03:04:38.578085 24548 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 691) took 4989 ms. Trace:
I20260925 03:04:38.578395 24548 rpcz_store.cc:276] 0925 03:04:33.588713 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:38.528205 (+4939492us) service_pool.cc:227] Handling call
0925 03:04:38.540026 (+ 11821us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:38.577843 (+ 37817us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:38.578029 (+ 186us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:38.579419 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5f9b503b0ab43999d582015a367441e" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
W20260925 03:04:38.543828 25220 debug-util.cc:398] Leaking SignalData structure 0x7b080033a1c0 after lost signal to thread 25224
I20260925 03:04:38.544106 24465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9c33fd100ff44c1a3247319a324a044" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:38.544945 25938 raft_consensus.cc:504] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } }
I20260925 03:04:38.550912 26199 raft_consensus.cc:482] T f22ddd7d1c444dc58f45a1fb0b3e02bc P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:38.551898 26239 leader_election.cc:288] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:38.552387 26086 leader_election.cc:288] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:38.552541 24616 raft_consensus.cc:2454] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 4.
W20260925 03:04:38.517510 24539 rpcz_store.cc:269] 0925 03:04:35.698615 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:38.298266 (+2599651us) service_pool.cc:227] Handling call
0925 03:04:38.357507 (+ 59241us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:38.553534 24467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6b6ec010caf4689b8c90449764f5cf9" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:38.479444 26240 raft_consensus.cc:482] T ac9357d144aa49789faaf31fe5e6bf89 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:38.585106 24465 raft_consensus.cc:2454] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 4.
I20260925 03:04:38.586024 26239 consensus_queue.cc:1035] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.002s
I20260925 03:04:38.587057 25938 leader_election.cc:288] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:38.580662 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c88cbd39e01f44628a094af516be2833" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:38.587416 24467 raft_consensus.cc:2454] T e6b6ec010caf4689b8c90449764f5cf9 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 3.
I20260925 03:04:38.589448 24548 raft_consensus.cc:2354] T c88cbd39e01f44628a094af516be2833 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 4. Current term is 5.
I20260925 03:04:38.589963 24467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43366a0a2f9d4344bf4e37ffd290ef8a" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:38.590723 24548 raft_consensus.cc:1261] T 550118bc1a0641ce9803bdbf43f2069f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260925 03:04:38.587026 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7aa47c48404465b94a36fd819fe3c7a" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:38.591517 24548 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 1250) took 4800 ms. Trace:
I20260925 03:04:38.591883 24539 raft_consensus.cc:2454] T a7aa47c48404465b94a36fd819fe3c7a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 2.
I20260925 03:04:38.585451 26199 raft_consensus.cc:504] T f22ddd7d1c444dc58f45a1fb0b3e02bc P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:38.593505 24539 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 700) took 3213 ms. Trace:
I20260925 03:04:38.593843 24539 rpcz_store.cc:276] 0925 03:04:35.379917 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:38.592652 (+3212735us) service_pool.cc:227] Handling call
0925 03:04:38.592994 (+ 342us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:38.593175 (+ 181us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:38.593203 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:38.593247 (+ 44us) raft_consensus.cc:1631] Marking committed up to 2
0925 03:04:38.593284 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:38.593338 (+ 54us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:38.593444 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:38.594758 26199 leader_election.cc:288] T f22ddd7d1c444dc58f45a1fb0b3e02bc P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:38.590773 24467 raft_consensus.cc:2379] T 43366a0a2f9d4344bf4e37ffd290ef8a P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c in current term 2: Already voted for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in this term.
I20260925 03:04:38.596009 24467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21c43af5e6844fdaacc8c625fa01fd8d" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:38.596719 24467 raft_consensus.cc:2379] T 21c43af5e6844fdaacc8c625fa01fd8d P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c in current term 2: Already voted for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in this term.
I20260925 03:04:38.592219 24465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60d45b7d467d4e1ab92367f6d7d52b15" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:38.597612 24467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4ebd2b043024a548005c8b6b00b0449" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:38.598201 24467 raft_consensus.cc:2454] T b4ebd2b043024a548005c8b6b00b0449 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 2.
I20260925 03:04:38.598325 24465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b627ecb4a5384ef392aa695e06599c39" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:38.598842 24467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5332a5d329493991dff994e70ea92c" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
W20260925 03:04:38.528625 24466 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 770) took 3078 ms (client timeout 3199 ms). Trace:
I20260925 03:04:38.562559 24545 rpcz_store.cc:276] 0925 03:04:33.634474 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:38.558320 (+4923846us) service_pool.cc:227] Handling call
0925 03:04:38.558610 (+ 290us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:38.558755 (+ 145us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:38.558776 (+ 21us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:38.558812 (+ 36us) raft_consensus.cc:1631] Marking committed up to 2
0925 03:04:38.558837 (+ 25us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:38.558881 (+ 44us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:38.558971 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:38.568311 24540 rpcz_store.cc:276] 0925 03:04:33.619388 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:38.528539 (+4909151us) service_pool.cc:227] Handling call
0925 03:04:38.532237 (+ 3698us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:38.532396 (+ 159us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:38.532420 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:38.532458 (+ 38us) raft_consensus.cc:1631] Marking committed up to 1
0925 03:04:38.532486 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:38.532533 (+ 47us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:38.532624 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260925 03:04:38.599426 24466 rpcz_store.cc:269] 0925 03:04:35.450690 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:38.477047 (+3026357us) service_pool.cc:227] Handling call
0925 03:04:38.528573 (+ 51526us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:38.600394 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b030d84c670b48d5bff337ccc0c77626" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:38.569226 26257 raft_consensus.cc:482] T 3d89396576884f7c8901dba54f783947 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:38.601135 24466 raft_consensus.cc:2379] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 5: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:04:38.601027 26257 raft_consensus.cc:504] T 3d89396576884f7c8901dba54f783947 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:38.601922 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:38.602461 24466 raft_consensus.cc:2454] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 4.
W20260925 03:04:38.540343 24541 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36786 (request call id 709) took 3153 ms (client timeout 3273 ms). Trace:
I20260925 03:04:38.572749 24472 raft_consensus.cc:2379] T 293ef69caf514ac98510a68495f54eda P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 3: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:04:38.603225 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc2df40db08649c9a75e90000b878559" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:38.603984 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5f9b503b0ab43999d582015a367441e" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:38.604210 24466 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "21c43af5e6844fdaacc8c625fa01fd8d"
dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
mode: GRACEFUL
new_leader_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
from {username='slave'} at 127.0.0.1:38832
I20260925 03:04:38.544951 24586 leader_election.cc:302] T c88cbd39e01f44628a094af516be2833 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:38.587859 26240 raft_consensus.cc:504] T ac9357d144aa49789faaf31fe5e6bf89 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:38.604712 24472 raft_consensus.cc:2379] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 5: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:04:38.606878 24466 raft_consensus.cc:595] T 21c43af5e6844fdaacc8c625fa01fd8d P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Received request to transfer leadership to TS a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:38.599068 24465 raft_consensus.cc:2454] T b627ecb4a5384ef392aa695e06599c39 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 4.
I20260925 03:04:38.600493 24540 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 707) took 3216 ms. Trace:
I20260925 03:04:38.569998 24542 raft_consensus.cc:1261] T 37a157996e944db2906110d87ecb0c54 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260925 03:04:38.605120 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 60267.7us
I20260925 03:04:38.610061 24586 leader_election.cc:302] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
W20260925 03:04:38.610227 24514 connection.cc:232] RPC call timeout handler was delayed by -2.3812210260002757s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15960000000000002s for a response
I20260925 03:04:38.610846 24546 raft_consensus.cc:3037] T ac9357d144aa49789faaf31fe5e6bf89 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:38.609421 24542 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 1245) took 4933 ms. Trace:
I20260925 03:04:38.608799 24540 rpcz_store.cc:276] 0925 03:04:35.384020 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:38.599780 (+3215760us) service_pool.cc:227] Handling call
0925 03:04:38.600046 (+ 266us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:38.600203 (+ 157us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:38.600228 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:38.600265 (+ 37us) raft_consensus.cc:1631] Marking committed up to 2
0925 03:04:38.600292 (+ 27us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:38.600338 (+ 46us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:38.600431 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:38.611614 24542 rpcz_store.cc:276] 0925 03:04:33.675916 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:38.569241 (+4893325us) service_pool.cc:227] Handling call
0925 03:04:38.569632 (+ 391us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:38.609150 (+ 39518us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:38.609364 (+ 214us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260925 03:04:38.608975 24464 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1202) took 2784 ms (client timeout 2738 ms). Trace:
W20260925 03:04:38.612936 24464 rpcz_store.cc:269] 0925 03:04:35.824363 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:38.476238 (+2651875us) service_pool.cc:227] Handling call
0925 03:04:38.608896 (+132658us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:38.591892 24548 rpcz_store.cc:276] 0925 03:04:33.791012 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:38.590062 (+4799050us) service_pool.cc:227] Handling call
0925 03:04:38.590391 (+ 329us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:38.591242 (+ 851us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:38.591453 (+ 211us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260925 03:04:38.606655 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 272935us
W20260925 03:04:38.614243 24588 connection.cc:232] RPC call timeout handler was delayed by -2.2310991899994406s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1626s for a response
W20260925 03:04:38.614488 24588 connection.cc:232] RPC call timeout handler was delayed by -2.2171634200003609s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.163s for a response
W20260925 03:04:38.614693 24588 connection.cc:232] RPC call timeout handler was delayed by -2.1113097010002093s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2015s for a response
W20260925 03:04:38.614938 24588 connection.cc:232] RPC call timeout handler was delayed by -2.0996097010011s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2028s for a response
W20260925 03:04:38.615288 24588 connection.cc:232] RPC call timeout handler was delayed by -2.0519097010001133s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2081s for a response
W20260925 03:04:38.615475 24588 connection.cc:232] RPC call timeout handler was delayed by -2.0393097010000929s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20950000000000002s for a response
W20260925 03:04:38.615713 24588 connection.cc:232] RPC call timeout handler was delayed by -2.036609701000998s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2098s for a response
W20260925 03:04:38.615927 24588 connection.cc:232] RPC call timeout handler was delayed by -1.9988097010009369s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21400000000000002s for a response
W20260925 03:04:38.616137 24588 connection.cc:232] RPC call timeout handler was delayed by -1.9583097009999619s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21850000000000003s for a response
W20260925 03:04:38.616334 24588 connection.cc:232] RPC call timeout handler was delayed by -1.8719097009998222s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22810000000000002s for a response
W20260925 03:04:38.616539 24588 connection.cc:232] RPC call timeout handler was delayed by -1.7234097010004916s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.24460000000000004s for a response
W20260925 03:04:38.616809 24588 connection.cc:232] RPC call timeout handler was delayed by -1.6766991900003632s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2242s for a response
W20260925 03:04:38.617163 24588 connection.cc:232] RPC call timeout handler was delayed by -1.2806097009997757s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2938s for a response
W20260925 03:04:38.617393 24588 connection.cc:232] RPC call timeout handler was delayed by -1.0754097010012629s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3166s for a response
W20260925 03:04:38.617590 24588 connection.cc:232] RPC call timeout handler was delayed by -1.0066237179999007s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17010000000000003s for a response
W20260925 03:04:38.619550 24588 leader_election.cc:334] T a7aa47c48404465b94a36fd819fe3c7a P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187): Timed out: RequestConsensusVote RPC to 127.22.193.1:42187 timed out after 1.630s (SENT)
I20260925 03:04:38.607245 24466 raft_consensus.cc:603] T 21c43af5e6844fdaacc8c625fa01fd8d P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260925 03:04:38.543568 26246 leader_election.cc:288] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:38.607999 25198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15861cd28e354598a3c6bfc4337c34bb" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:38.543533 26209 leader_election.cc:288] T 9e8af2641e124f9589c5ccfdac785f63 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:38.622972 24586 leader_election.cc:302] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:38.623446 24464 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 746) took 4239 ms. Trace:
W20260925 03:04:38.607228 24569 debug-util.cc:398] Leaking SignalData structure 0x7b080036e1a0 after lost signal to thread 25224
I20260925 03:04:38.621666 25198 raft_consensus.cc:2396] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
W20260925 03:04:38.610885 24514 connection.cc:232] RPC call timeout handler was delayed by -2.033821026001533s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19820000000000002s for a response
I20260925 03:04:38.613571 24545 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 716) took 3040 ms. Trace:
I20260925 03:04:38.603247 26257 leader_election.cc:288] T 3d89396576884f7c8901dba54f783947 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:38.576546 24544 raft_consensus.cc:2379] T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 in current term 5: Already voted for candidate eeab93abfdb141a895002ec9f312296c in this term.
I20260925 03:04:38.623771 24464 rpcz_store.cc:276] 0925 03:04:34.383914 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:38.613598 (+4229684us) service_pool.cc:227] Handling call
0925 03:04:38.622955 (+ 9357us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:38.623168 (+ 213us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:38.623191 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:38.623229 (+ 38us) raft_consensus.cc:1631] Marking committed up to 3
0925 03:04:38.623255 (+ 26us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:38.623303 (+ 48us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:38.623397 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:38.537755 26068 consensus_queue.cc:1035] T 367277daf8d04778b3fa4ed94d71580d P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.072s
I20260925 03:04:38.626609 24470 raft_consensus.cc:1226] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 4. Current term is 5. Ops: []
I20260925 03:04:38.626811 24472 raft_consensus.cc:1226] T 442d4a29b1d743599f4b6b4871e26668 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 4. Current term is 5. Ops: []
I20260925 03:04:38.570467 25194 raft_consensus.cc:1261] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43 [term 5 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260925 03:04:38.628997 24545 rpcz_store.cc:276] 0925 03:04:35.572973 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:38.612899 (+3039926us) service_pool.cc:227] Handling call
0925 03:04:38.613152 (+ 253us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:38.613306 (+ 154us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:38.613330 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:38.613368 (+ 38us) raft_consensus.cc:1631] Marking committed up to 1
0925 03:04:38.613394 (+ 26us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:38.613442 (+ 48us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:38.613529 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:38.639298 24545 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 719) took 3065 ms. Trace:
I20260925 03:04:38.633550 24472 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 745) took 4259 ms. Trace:
I20260925 03:04:38.632082 24548 raft_consensus.cc:1226] T 442d4a29b1d743599f4b6b4871e26668 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 4. Current term is 5. Ops: []
I20260925 03:04:38.630599 24586 leader_election.cc:302] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 1bc1f2e62ab44af995165bd5116a9d43, a04a37fc4aa7460cbfdbeeb4e451e1f5
W20260925 03:04:38.628932 24514 connection.cc:232] RPC call timeout handler was delayed by -1.8376210260012158s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22000000000000003s for a response
I20260925 03:04:38.636421 24467 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 742) took 4322 ms. Trace:
I20260925 03:04:38.641179 24548 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 718) took 3067 ms. Trace:
W20260925 03:04:38.641577 24514 connection.cc:232] RPC call timeout handler was delayed by -1.7988767940005346s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1704s for a response
I20260925 03:04:38.640268 24472 rpcz_store.cc:276] 0925 03:04:34.374056 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:38.607308 (+4233252us) service_pool.cc:227] Handling call
0925 03:04:38.626654 (+ 19346us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:38.633315 (+ 6661us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:38.633499 (+ 184us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260925 03:04:38.642005 24514 connection.cc:232] RPC call timeout handler was delayed by -1.3470767940016231s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22060000000000002s for a response
I20260925 03:04:38.635785 24546 raft_consensus.cc:2454] T ac9357d144aa49789faaf31fe5e6bf89 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 2.
I20260925 03:04:38.641916 24548 rpcz_store.cc:276] 0925 03:04:35.573623 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:38.614047 (+3040424us) service_pool.cc:227] Handling call
0925 03:04:38.631899 (+ 17852us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:38.640922 (+ 9023us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:38.641125 (+ 203us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:38.639679 24545 rpcz_store.cc:276] 0925 03:04:35.573895 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:38.638567 (+3064672us) service_pool.cc:227] Handling call
0925 03:04:38.638836 (+ 269us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:38.639000 (+ 164us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:38.639023 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:38.639065 (+ 42us) raft_consensus.cc:1631] Marking committed up to 3
0925 03:04:38.639094 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:38.639148 (+ 54us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:38.639247 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:38.641611 24467 rpcz_store.cc:276] 0925 03:04:34.314180 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:38.605788 (+4291608us) service_pool.cc:227] Handling call
0925 03:04:38.636106 (+ 30318us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:38.636226 (+ 120us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:38.636240 (+ 14us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:38.636265 (+ 25us) raft_consensus.cc:1631] Marking committed up to 3
0925 03:04:38.636280 (+ 15us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:38.636310 (+ 30us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:38.636374 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:38.645565 24586 leader_election.cc:302] T c85f5d1d3f004cf5817bb594fd8e6469 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:38.646301 24586 leader_election.cc:302] T 71f461a6e2ff434993b78a2d077f0abd P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters: 1bc1f2e62ab44af995165bd5116a9d43
I20260925 03:04:38.611349 26245 raft_consensus.cc:2781] T c88cbd39e01f44628a094af516be2833 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Leader election won for term 5
I20260925 03:04:38.649878 24464 raft_consensus.cc:1226] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 3. Current term is 4. Ops: []
I20260925 03:04:38.623008 26247 raft_consensus.cc:2781] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Leader election won for term 5
I20260925 03:04:38.622841 26181 raft_consensus.cc:482] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1bc1f2e62ab44af995165bd5116a9d43)
I20260925 03:04:38.655655 24548 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 723) took 3067 ms. Trace:
I20260925 03:04:38.658119 24548 rpcz_store.cc:276] 0925 03:04:35.588044 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:38.644418 (+3056374us) service_pool.cc:227] Handling call
0925 03:04:38.655198 (+ 10780us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:38.655364 (+ 166us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:38.655386 (+ 22us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:38.655424 (+ 38us) raft_consensus.cc:1631] Marking committed up to 2
0925 03:04:38.655452 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:38.655500 (+ 48us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:38.655604 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:38.656008 24464 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55560 (request call id 1178) took 3630 ms. Trace:
I20260925 03:04:38.638940 26246 raft_consensus.cc:2781] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader election won for term 3
I20260925 03:04:38.606384 25804 raft_consensus.cc:504] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:38.567937 25897 raft_consensus.cc:504] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:38.665895 24467 raft_consensus.cc:1226] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 3. Current term is 4. Ops: []
I20260925 03:04:38.666620 24467 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55560 (request call id 1183) took 3640 ms. Trace:
I20260925 03:04:38.666947 24467 rpcz_store.cc:276] 0925 03:04:35.026572 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:38.648136 (+3621564us) service_pool.cc:227] Handling call
0925 03:04:38.665715 (+ 17579us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:38.666359 (+ 644us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:38.666563 (+ 204us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260925 03:04:38.606310 24541 rpcz_store.cc:269] 0925 03:04:35.384562 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:38.303958 (+2919396us) service_pool.cc:227] Handling call
0925 03:04:38.538393 (+234435us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:38.662535 26240 leader_election.cc:288] T ac9357d144aa49789faaf31fe5e6bf89 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:38.660355 24464 rpcz_store.cc:276] 0925 03:04:35.025153 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:38.645025 (+3619872us) service_pool.cc:227] Handling call
0925 03:04:38.649721 (+ 4696us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:38.655779 (+ 6058us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:38.655939 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1972736}
W20260925 03:04:38.667387 24436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 353223us
I20260925 03:04:38.627604 26256 raft_consensus.cc:482] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:38.671197 24472 raft_consensus.cc:1226] T ceab6b0fccd54e909a64f66fe81a9a1f P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 3. Current term is 4. Ops: []
I20260925 03:04:38.678890 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcf6e6a422d4d198d4fbb0f7d3d845d" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:38.679867 24615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442d4a29b1d743599f4b6b4871e26668" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:38.681188 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5332a5d329493991dff994e70ea92c" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:38.686702 24621 raft_consensus.cc:2379] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 5: Already voted for candidate eeab93abfdb141a895002ec9f312296c in this term.
I20260925 03:04:38.648515 24465 raft_consensus.cc:1226] T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 LEADER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 3. Current term is 4. Ops: []
I20260925 03:04:38.681394 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06f4a7eaf6534e4986c9fd35911038e4" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:38.683192 24436 leader_election.cc:302] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:38.649114 26083 raft_consensus.cc:2726] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260925 03:04:38.607604 26107 raft_consensus.cc:504] T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:38.684916 24514 leader_election.cc:302] T 550118bc1a0641ce9803bdbf43f2069f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:38.685917 24464 raft_consensus.cc:1226] T 37a157996e944db2906110d87ecb0c54 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 LEADER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 3. Current term is 4. Ops: []
W20260925 03:04:38.684736 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 117467us
I20260925 03:04:38.655220 26248 raft_consensus.cc:2781] T c85f5d1d3f004cf5817bb594fd8e6469 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260925 03:04:38.690222 24472 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55560 (request call id 1180) took 3664 ms. Trace:
I20260925 03:04:38.702186 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74443a0a1dc34d88a2f42e3b722db405" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:38.702675 24472 rpcz_store.cc:276] 0925 03:04:35.025736 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:38.648179 (+3622443us) service_pool.cc:227] Handling call
0925 03:04:38.671038 (+ 22859us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:38.689995 (+ 18957us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:38.690171 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:38.690550 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f22ddd7d1c444dc58f45a1fb0b3e02bc" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:38.656159 25942 raft_consensus.cc:2781] T 71f461a6e2ff434993b78a2d077f0abd P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260925 03:04:38.706190 25942 raft_consensus.cc:482] T 71f461a6e2ff434993b78a2d077f0abd P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting leader election (detected failure of leader 1bc1f2e62ab44af995165bd5116a9d43)
I20260925 03:04:38.706911 24618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ac466793d094325ace224708b2158ed" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:38.707409 24514 leader_election.cc:302] T 25102ba08e4e49388f6176badcd10a25 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:04:38.707587 24618 raft_consensus.cc:2454] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 3.
I20260925 03:04:38.703796 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37a157996e944db2906110d87ecb0c54" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:38.704267 24471 raft_consensus.cc:2396] T f22ddd7d1c444dc58f45a1fb0b3e02bc P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260925 03:04:38.704809 24464 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55560 (request call id 1185) took 3677 ms. Trace:
I20260925 03:04:38.711058 24464 rpcz_store.cc:276] 0925 03:04:35.027117 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:38.682624 (+3655507us) service_pool.cc:227] Handling call
0925 03:04:38.685736 (+ 3112us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:38.704532 (+ 18796us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:38.704750 (+ 218us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:38.706617 25942 raft_consensus.cc:3037] T 71f461a6e2ff434993b78a2d077f0abd P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:38.709728 24436 leader_election.cc:302] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:38.712432 24614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e5819c15eac4544a57e5ee6c9d4f117" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:38.706445 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceab6b0fccd54e909a64f66fe81a9a1f" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:38.681730 26246 raft_consensus.cc:686] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c [term 3 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
W20260925 03:04:38.714675 24437 connection.cc:232] RPC call timeout handler was delayed by -2.4316766829997505s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2818s for a response
I20260925 03:04:38.715858 24615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e725b82985746b39c0cc8f671c95353" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:38.682171 26255 raft_consensus.cc:2781] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader election won for term 5
I20260925 03:04:38.709992 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c88cbd39e01f44628a094af516be2833" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
W20260925 03:04:38.718389 24437 connection.cc:232] RPC call timeout handler was delayed by -1.9743258409998816s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17170000000000002s for a response
W20260925 03:04:38.719066 24437 connection.cc:232] RPC call timeout handler was delayed by -1.9248258409988921s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17720000000000002s for a response
I20260925 03:04:38.720381 24618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4abe0ea757e4886be4945c5001e529a" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
W20260925 03:04:38.720263 24437 connection.cc:232] RPC call timeout handler was delayed by -1.8393258409996633s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1867s for a response
I20260925 03:04:38.719623 24615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48ca6ccd98194cf2af7b5278d15bf4ea" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
W20260925 03:04:38.721518 24437 connection.cc:232] RPC call timeout handler was delayed by -1.8339258409996546s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18730000000000002s for a response
I20260925 03:04:38.721530 24618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b8e0d260a884ecdb20f79de46c36b19" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
W20260925 03:04:38.722635 24437 connection.cc:232] RPC call timeout handler was delayed by -1.8114258410005277s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1898s for a response
I20260925 03:04:38.724217 24511 leader_election.cc:302] T 8ac466793d094325ace224708b2158ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters: 1bc1f2e62ab44af995165bd5116a9d43
W20260925 03:04:38.724169 24437 connection.cc:232] RPC call timeout handler was delayed by -1.5234258410000621s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2218s for a response
I20260925 03:04:38.724506 24618 raft_consensus.cc:2354] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 3. Current term is 4.
W20260925 03:04:38.725389 24437 connection.cc:232] RPC call timeout handler was delayed by -1.50362584100003s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22400000000000003s for a response
I20260925 03:04:38.723443 24615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60d45b7d467d4e1ab92367f6d7d52b15" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
W20260925 03:04:38.726593 24437 connection.cc:232] RPC call timeout handler was delayed by -1.2352804669990292s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16820000000000002s for a response
I20260925 03:04:38.726127 24618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35992316cb8d4c028333388eff02ecde" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
W20260925 03:04:38.727655 24437 connection.cc:232] RPC call timeout handler was delayed by -1.218180466999911s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17010000000000003s for a response
I20260925 03:04:38.728876 24618 raft_consensus.cc:2454] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 4.
W20260925 03:04:38.728858 24437 connection.cc:232] RPC call timeout handler was delayed by -1.1668804669989186s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1758s for a response
W20260925 03:04:38.730443 24437 connection.cc:232] RPC call timeout handler was delayed by -1.1506804669988924s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1776s for a response
I20260925 03:04:38.731106 26245 raft_consensus.cc:686] T c88cbd39e01f44628a094af516be2833 P eeab93abfdb141a895002ec9f312296c [term 5 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
W20260925 03:04:38.731249 24437 connection.cc:232] RPC call timeout handler was delayed by -1.1272804669988545s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18020000000000003s for a response
I20260925 03:04:38.731907 24618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f22ddd7d1c444dc58f45a1fb0b3e02bc" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:38.732656 26076 leader_election.cc:288] T 7e49248643df4131a1c42e74244e05c2 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 5 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
W20260925 03:04:38.735534 24437 connection.cc:232] RPC call timeout handler was delayed by -1.1110804669988283s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18200000000000002s for a response
I20260925 03:04:38.734865 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9357d144aa49789faaf31fe5e6bf89" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
W20260925 03:04:38.735872 24437 connection.cc:232] RPC call timeout handler was delayed by -1.0815258410002571s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.27090000000000003s for a response
W20260925 03:04:38.736569 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 47158.3us
I20260925 03:04:38.736744 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b030d84c670b48d5bff337ccc0c77626" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:38.738078 24470 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 741) took 4424 ms. Trace:
I20260925 03:04:38.738502 24470 rpcz_store.cc:276] 0925 03:04:34.313463 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:38.604950 (+4291487us) service_pool.cc:227] Handling call
0925 03:04:38.624852 (+ 19902us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:38.645046 (+ 20194us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:38.737850 (+ 92804us) spinlock_profiling.cc:243] Waited 19.6 ms on lock 0x7b540011add0. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764cfa09 00007f437b4112e9 00007f4374af7684 00007f4374af7612 00007f4374af7541 00007f4374af74bc 00007f4374af6772 00007f43748fa94c 00007f43748f9d86 00007f43748f972f
0925 03:04:38.738005 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":44983040}
I20260925 03:04:38.739114 24586 leader_election.cc:302] T cc2df40db08649c9a75e90000b878559 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:04:38.739975 24470 raft_consensus.cc:1226] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 3. Current term is 4. Ops: []
I20260925 03:04:38.691612 25982 leader_election.cc:288] T b1e85248d3b04b979dd080caeeb02fdb P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:38.742030 24586 leader_election.cc:302] T edcf6e6a422d4d198d4fbb0f7d3d845d P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:38.706533 25804 leader_election.cc:288] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:38.743184 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9c33fd100ff44c1a3247319a324a044" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:38.750283 24472 raft_consensus.cc:3037] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:38.752442 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e54ca1757ee4e42956a9cebbeca4f62" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:38.740619 24470 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55560 (request call id 1191) took 3711 ms. Trace:
I20260925 03:04:38.745599 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b8e0d260a884ecdb20f79de46c36b19" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:38.745997 24511 leader_election.cc:302] T a4abe0ea757e4886be4945c5001e529a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 1bc1f2e62ab44af995165bd5116a9d43, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:38.610875 26244 raft_consensus.cc:504] T 367277daf8d04778b3fa4ed94d71580d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } }
I20260925 03:04:38.696563 26209 raft_consensus.cc:2781] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260925 03:04:38.778060 24437 leader_election.cc:302] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:04:38.700862 24613 raft_consensus.cc:2454] T edcf6e6a422d4d198d4fbb0f7d3d845d P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 1.
I20260925 03:04:38.702579 26248 raft_consensus.cc:482] T c85f5d1d3f004cf5817bb594fd8e6469 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting leader election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:38.753485 24470 rpcz_store.cc:276] 0925 03:04:35.028788 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:38.739432 (+3710644us) service_pool.cc:227] Handling call
0925 03:04:38.739799 (+ 367us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:38.740375 (+ 576us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:38.740567 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:38.671020 26181 raft_consensus.cc:504] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:38.693488 26175 raft_consensus.cc:504] T 37a157996e944db2906110d87ecb0c54 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:38.779475 24471 raft_consensus.cc:3037] T 7b8e0d260a884ecdb20f79de46c36b19 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:38.777964 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d89396576884f7c8901dba54f783947" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:38.733119 25938 raft_consensus.cc:482] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:38.733471 24465 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55560 (request call id 1172) took 4335 ms. Trace:
I20260925 03:04:38.741706 25957 raft_consensus.cc:2781] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260925 03:04:38.733088 26240 raft_consensus.cc:2726] T 550118bc1a0641ce9803bdbf43f2069f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260925 03:04:38.780814 26209 raft_consensus.cc:482] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:38.793246 24470 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "48ca6ccd98194cf2af7b5278d15bf4ea"
dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
mode: GRACEFUL
new_leader_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
from {username='slave'} at 127.0.0.1:38832
I20260925 03:04:38.827345 24472 raft_consensus.cc:2454] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 5.
I20260925 03:04:38.714293 26199 consensus_queue.cc:1035] T 550118bc1a0641ce9803bdbf43f2069f P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.069s
I20260925 03:04:38.779203 26257 raft_consensus.cc:2781] T 25102ba08e4e49388f6176badcd10a25 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260925 03:04:38.833498 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 126141us
W20260925 03:04:38.878947 26115 consensus_meta.cc:287] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c: Time spent flushing consensus metadata: real 0.578s user 0.006s sys 0.003s
I20260925 03:04:38.833514 24620 raft_consensus.cc:2379] T 37a157996e944db2906110d87ecb0c54 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 5: Already voted for candidate eeab93abfdb141a895002ec9f312296c in this term.
I20260925 03:04:38.782773 24511 leader_election.cc:398] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 pre-election: Vote denied by peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105) with higher term. Message: Invalid argument: T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 3. Current term is 4.
I20260925 03:04:38.783095 25804 raft_consensus.cc:2726] T a4abe0ea757e4886be4945c5001e529a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260925 03:04:38.780731 26239 raft_consensus.cc:2781] T 8ac466793d094325ace224708b2158ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260925 03:04:38.743788 26196 raft_consensus.cc:504] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:38.693712 26256 raft_consensus.cc:504] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:38.869972 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:38.788933 26248 raft_consensus.cc:3037] T c85f5d1d3f004cf5817bb594fd8e6469 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Advancing to term 4
W20260925 03:04:38.838784 26258 log.cc:924] Time spent T a7aa47c48404465b94a36fd819fe3c7a P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.089s user 0.000s sys 0.001s
I20260925 03:04:38.882351 26255 raft_consensus.cc:686] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 LEADER]: Becoming Leader. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:04:38.837244 25957 raft_consensus.cc:482] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:38.925472 24470 raft_consensus.cc:595] T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Received request to transfer leadership to TS a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:38.866891 26209 raft_consensus.cc:3037] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:39.056059 24511 leader_election.cc:401] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20260925 03:04:38.988595 24588 leader_election.cc:302] T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:39.067163 24514 leader_election.cc:302] T 11bb46bce3564841b7674e6a693bd2e6 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
W20260925 03:04:39.067332 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 78860.6us
I20260925 03:04:38.939908 24471 raft_consensus.cc:2454] T 7b8e0d260a884ecdb20f79de46c36b19 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 4.
I20260925 03:04:39.058292 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15861cd28e354598a3c6bfc4337c34bb" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:39.066469 24470 raft_consensus.cc:603] T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260925 03:04:39.065367 24617 raft_consensus.cc:2454] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 4.
I20260925 03:04:39.071183 24540 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 712) took 3684 ms. Trace:
I20260925 03:04:38.866632 25938 raft_consensus.cc:504] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:39.072435 24540 rpcz_store.cc:276] 0925 03:04:35.386959 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:38.612437 (+3225478us) service_pool.cc:227] Handling call
0925 03:04:38.612764 (+ 327us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:38.828674 (+215910us) raft_consensus.cc:1520] Early marking committed up to index 4
0925 03:04:38.883029 (+ 54355us) log.cc:842] Serialized 14 byte log entry
0925 03:04:39.055545 (+172516us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:39.055598 (+ 53us) raft_consensus.cc:1631] Marking committed up to 4
0925 03:04:39.055639 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:39.068957 (+ 13318us) spinlock_profiling.cc:243] Waited 216 ms on lock 0x7b540036bd54. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d0f51 00007f43764cf99e 00007f437b4112e9 00007f4374af7684 00007f4374af7612 00007f4374af7541 00007f4374af74bc 00007f4374af6772 00007f43748fa94c 00007f43748f9d86
0925 03:04:39.069022 (+ 65us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:39.071039 (+ 2017us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":155934,"raft.queue_time_us":114351,"spinlock_wait_cycles":496004480,"thread_start_us":51041,"threads_started":2}
I20260925 03:04:39.071301 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5aec38e66a947c681e6c2a72b05e6ca" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
W20260925 03:04:38.936348 26138 consensus_meta.cc:287] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent flushing consensus metadata: real 0.631s user 0.010s sys 0.005s
I20260925 03:04:39.079716 24615 raft_consensus.cc:2379] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 5: Already voted for candidate eeab93abfdb141a895002ec9f312296c in this term.
I20260925 03:04:38.929466 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f1fac028a144058b92331c0d9b44ebf" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:39.080323 24540 raft_consensus.cc:1261] T 7993f9c3e2a44f408bbf5d1e395e2306 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260925 03:04:39.077986 24471 raft_consensus.cc:2454] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 5.
W20260925 03:04:39.081569 24511 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 298888us
I20260925 03:04:39.086380 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37a157996e944db2906110d87ecb0c54" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:38.828318 24465 rpcz_store.cc:276] 0925 03:04:34.398456 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:38.631212 (+4232756us) service_pool.cc:227] Handling call
0925 03:04:38.631551 (+ 339us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:38.694014 (+ 62463us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:38.717765 (+ 23751us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:39.035094 25196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "293ef69caf514ac98510a68495f54eda" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:39.055092 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71f461a6e2ff434993b78a2d077f0abd" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:39.091943 24511 leader_election.cc:302] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:39.098891 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c23337a7cb74184a0e57b7095cb6a11" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:38.931684 26241 leader_election.cc:288] T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:39.036635 25195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1e85248d3b04b979dd080caeeb02fdb" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:39.013104 25193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:38.933961 25979 leader_election.cc:288] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:38.993582 26107 leader_election.cc:288] T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 election: Requested vote from peers eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:38.942515 25897 leader_election.cc:288] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:39.107650 24472 raft_consensus.cc:2379] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c in current term 5: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:04:39.069128 26239 raft_consensus.cc:482] T 8ac466793d094325ace224708b2158ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260925 03:04:38.937369 24616 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38564 (request call id 1217) took 4443 ms (client timeout 1912 ms). Trace:
W20260925 03:04:39.137859 24616 rpcz_store.cc:269] 0925 03:04:34.494403 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:35.567044 (+1072641us) service_pool.cc:227] Handling call
0925 03:04:38.937189 (+3370145us) spinlock_profiling.cc:243] Waited 2.98 s on lock 0x7b5400217fd4. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d42ae 00007f437b411c27 00007f4374af9c04 00007f4374af9b92 00007f4374af9ac1 00007f4374af9a3c 00007f4374af8cf2 00007f43748fa94c 00007f43748f9d86 00007f43748f972f
0925 03:04:38.937300 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6863683328}
I20260925 03:04:39.108497 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9357d144aa49789faaf31fe5e6bf89" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:39.138635 26239 raft_consensus.cc:3037] T 8ac466793d094325ace224708b2158ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:39.138948 25195 raft_consensus.cc:2396] T b1e85248d3b04b979dd080caeeb02fdb P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 for term 3 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260925 03:04:39.139241 24617 raft_consensus.cc:2379] T ac9357d144aa49789faaf31fe5e6bf89 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 2: Already voted for candidate eeab93abfdb141a895002ec9f312296c in this term.
I20260925 03:04:39.014737 25194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:39.140031 25938 leader_election.cc:288] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:39.078709 26138 raft_consensus.cc:504] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:39.140861 25938 raft_consensus.cc:2781] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260925 03:04:39.141207 24543 raft_consensus.cc:2379] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 in current term 5: Already voted for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in this term.
I20260925 03:04:39.141548 26241 raft_consensus.cc:2781] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260925 03:04:39.141965 26241 raft_consensus.cc:482] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:39.142287 26241 raft_consensus.cc:3037] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:39.143356 25193 raft_consensus.cc:2454] T 29637f5a4a504ce6ba29033ccd0cd716 P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 3.
I20260925 03:04:39.144344 25190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5aec38e66a947c681e6c2a72b05e6ca" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:39.145048 25190 raft_consensus.cc:2454] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 5.
I20260925 03:04:39.070605 25957 raft_consensus.cc:3037] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:04:39.146014 26271 consensus_queue.cc:1035] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260925 03:04:39.147034 26196 leader_election.cc:288] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:39.147387 26271 raft_consensus.cc:2781] T cc2df40db08649c9a75e90000b878559 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260925 03:04:39.147814 26271 raft_consensus.cc:482] T cc2df40db08649c9a75e90000b878559 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:38.943168 26257 raft_consensus.cc:482] T 25102ba08e4e49388f6176badcd10a25 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:39.037721 25191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6b6ec010caf4689b8c90449764f5cf9" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:39.075714 25804 raft_consensus.cc:2726] T 11bb46bce3564841b7674e6a693bd2e6 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260925 03:04:39.115852 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 49393.3us
I20260925 03:04:39.116464 24511 leader_election.cc:302] T edcf6e6a422d4d198d4fbb0f7d3d845d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:39.134754 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "293ef69caf514ac98510a68495f54eda" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:39.232350 24472 raft_consensus.cc:2454] T 293ef69caf514ac98510a68495f54eda P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 3.
W20260925 03:04:39.143466 24543 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 1275) took 3307 ms (client timeout 3136 ms). Trace:
I20260925 03:04:39.148134 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
W20260925 03:04:39.113058 26260 log.cc:924] Time spent T 1414dcfe53f64a78ae413531eb8ec117 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.212s user 0.001s sys 0.000s
I20260925 03:04:38.925747 26264 consensus_queue.cc:1035] T a4abe0ea757e4886be4945c5001e529a P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.243s
I20260925 03:04:39.087525 26086 raft_consensus.cc:2726] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105) with higher term. Message: Invalid argument: T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 3. Current term is 4.
I20260925 03:04:39.055800 26115 raft_consensus.cc:504] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:39.239599 26115 leader_election.cc:288] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:39.240607 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "293ef69caf514ac98510a68495f54eda" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:39.241331 24472 raft_consensus.cc:2454] T 293ef69caf514ac98510a68495f54eda P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 3.
I20260925 03:04:39.242235 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:39.242977 24472 raft_consensus.cc:2379] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1bc1f2e62ab44af995165bd5116a9d43 in current term 4: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:04:39.241042 24618 raft_consensus.cc:2379] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 5: Already voted for candidate eeab93abfdb141a895002ec9f312296c in this term.
I20260925 03:04:38.901670 26246 consensus_queue.cc:237] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:39.232419 24514 leader_election.cc:398] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187) with higher term. Message: Invalid argument: T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 3. Current term is 5.
W20260925 03:04:39.220093 26266 log.cc:924] Time spent T 367277daf8d04778b3fa4ed94d71580d P 1bc1f2e62ab44af995165bd5116a9d43: Append to log took a long time: real 0.176s user 0.000s sys 0.001s
I20260925 03:04:39.109057 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
W20260925 03:04:39.149523 24436 connection.cc:232] RPC call timeout handler was delayed by -3.3623933369999577s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19890000000000002s for a response
W20260925 03:04:39.247715 24436 connection.cc:232] RPC call timeout handler was delayed by -3.1364933370005019s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22400000000000003s for a response
W20260925 03:04:39.248055 24436 connection.cc:232] RPC call timeout handler was delayed by -2.611793337000563s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2823s for a response
W20260925 03:04:39.233507 24511 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 117110us
I20260925 03:04:39.141258 25938 raft_consensus.cc:482] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:39.216666 26257 raft_consensus.cc:3037] T 25102ba08e4e49388f6176badcd10a25 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:39.255254 24436 leader_election.cc:302] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5; no voters: eeab93abfdb141a895002ec9f312296c
W20260925 03:04:39.258623 24585 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.576s user 0.041s sys 0.084s
W20260925 03:04:39.143388 26268 log.cc:924] Time spent T b627ecb4a5384ef392aa695e06599c39 P 869b0a8575b54d08ae2817fad5dd42d5: Append to log took a long time: real 0.218s user 0.000s sys 0.000s
I20260925 03:04:39.140056 25194 raft_consensus.cc:2379] T 7e49248643df4131a1c42e74244e05c2 P 1bc1f2e62ab44af995165bd5116a9d43 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 in current term 5: Already voted for candidate 1bc1f2e62ab44af995165bd5116a9d43 in this term.
I20260925 03:04:39.216811 26271 raft_consensus.cc:3037] T cc2df40db08649c9a75e90000b878559 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:39.236701 26107 raft_consensus.cc:2781] T edcf6e6a422d4d198d4fbb0f7d3d845d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 03:04:39.063725 26245 consensus_queue.cc:237] T c88cbd39e01f44628a094af516be2833 P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:39.233130 24471 raft_consensus.cc:3037] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:39.245347 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c81a1b3b616492ab6c964f45af36077" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:39.136104 24541 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 729) took 3302 ms. Trace:
I20260925 03:04:39.247788 24470 raft_consensus.cc:2454] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 4.
W20260925 03:04:39.233647 24543 rpcz_store.cc:269] 0925 03:04:35.835605 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:38.314885 (+2479280us) service_pool.cc:227] Handling call
0925 03:04:39.143254 (+828369us) spinlock_profiling.cc:243] Waited 820 ms on lock 0x7b540034b2d4. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d42ae 00007f437b411c27 00007f4374af9c04 00007f4374af9b92 00007f4374af9ac1 00007f4374af9a3c 00007f4374af8cf2 00007f43748fa94c 00007f43748f9d86 00007f43748f972f
0925 03:04:39.143400 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1885338368}
W20260925 03:04:39.272959 24544 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.597s user 0.007s sys 0.004s
W20260925 03:04:39.277243 24544 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.607s user 0.007s sys 0.004s
I20260925 03:04:39.275719 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15861cd28e354598a3c6bfc4337c34bb" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:39.281970 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c81a1b3b616492ab6c964f45af36077" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:39.275256 24472 raft_consensus.cc:2454] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 3.
W20260925 03:04:39.229218 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 487148us
I20260925 03:04:39.257129 25938 raft_consensus.cc:3037] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:39.235617 26138 leader_election.cc:288] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:39.147738 26196 raft_consensus.cc:2781] T edcf6e6a422d4d198d4fbb0f7d3d845d P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260925 03:04:39.258971 24585 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.577s user 0.041s sys 0.084s
I20260925 03:04:39.236250 24469 raft_consensus.cc:2379] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 4: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:04:39.255986 24547 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 688) took 5675 ms. Trace:
W20260925 03:04:39.236830 26261 log.cc:924] Time spent T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.451s user 0.000s sys 0.001s
I20260925 03:04:39.265344 24514 leader_election.cc:302] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:39.266871 24436 leader_election.cc:302] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5; no voters:
I20260925 03:04:39.248661 24545 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 721) took 3674 ms. Trace:
I20260925 03:04:39.224411 25979 consensus_queue.cc:1046] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.045s
I20260925 03:04:39.283875 24540 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7993f9c3e2a44f408bbf5d1e395e2306"
dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
mode: GRACEFUL
new_leader_uuid: "eeab93abfdb141a895002ec9f312296c"
from {username='slave'} at 127.0.0.1:40158
I20260925 03:04:39.284191 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1e85248d3b04b979dd080caeeb02fdb" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:39.268011 26175 leader_election.cc:288] T 37a157996e944db2906110d87ecb0c54 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:39.298390 24540 raft_consensus.cc:595] T 7993f9c3e2a44f408bbf5d1e395e2306 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Received request to transfer leadership to TS eeab93abfdb141a895002ec9f312296c
I20260925 03:04:39.299116 24540 raft_consensus.cc:603] T 7993f9c3e2a44f408bbf5d1e395e2306 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260925 03:04:39.299417 24358 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: GetConsensusState request on kudu.consensus.ConsensusService from 127.0.0.1:55552 dropped due to backpressure. The service queue is full; it has 50 items. [suppressed 65 similar messages]
W20260925 03:04:39.286206 26247 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.580s user 0.053s sys 0.099s
I20260925 03:04:39.288615 24470 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 759) took 3840 ms. Trace:
I20260925 03:04:39.255642 25193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e068a0f80941dc9138cee063f53bab" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43"
I20260925 03:04:39.255918 25190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e92f377a79354a6ebb4eec67303c13e3" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:39.268311 26199 raft_consensus.cc:482] T 8e5819c15eac4544a57e5ee6c9d4f117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:39.293859 26196 raft_consensus.cc:482] T edcf6e6a422d4d198d4fbb0f7d3d845d P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Starting leader election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:39.255378 26256 leader_election.cc:288] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:39.296365 24514 leader_election.cc:302] T 420163a6cdce412ebc21c25fe3835638 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:04:39.294552 24547 rpcz_store.cc:276] 0925 03:04:33.580579 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:38.515522 (+4934943us) service_pool.cc:227] Handling call
0925 03:04:38.515949 (+ 427us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:04:38.671745 (+155796us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:38.671763 (+ 18us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:04:38.692910 (+ 21147us) log.cc:842] Serialized 26 byte log entry
0925 03:04:38.834162 (+141252us) raft_consensus.cc:1631] Marking committed up to 2
0925 03:04:38.834197 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:38.834261 (+ 64us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:04:39.255767 (+421506us) raft_consensus.cc:1671] finished
0925 03:04:39.255787 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:39.255929 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":69490,"raft.run_cpu_time_us":104,"raft.run_wall_time_us":104,"thread_start_us":244495,"threads_started":2,"wal-append.queue_time_us":382486}
I20260925 03:04:39.267459 24465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:39.298915 24472 raft_consensus.cc:2454] T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 2.
W20260925 03:04:39.235744 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 457701us
I20260925 03:04:39.295979 24545 rpcz_store.cc:276] 0925 03:04:35.574502 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:38.643988 (+3069486us) service_pool.cc:227] Handling call
0925 03:04:38.655812 (+ 11824us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:38.655999 (+ 187us) raft_consensus.cc:1520] Early marking committed up to index 2
0925 03:04:38.779273 (+123274us) log.cc:842] Serialized 14 byte log entry
0925 03:04:39.248100 (+468827us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:39.248209 (+ 109us) raft_consensus.cc:1631] Marking committed up to 2
0925 03:04:39.248277 (+ 68us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:39.248365 (+ 88us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:39.248558 (+ 193us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":481358,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":40,"raft.queue_time_us":2786,"raft.run_cpu_time_us":81,"raft.run_wall_time_us":81,"thread_start_us":582539,"threads_started":2,"wal-append.queue_time_us":463542}
I20260925 03:04:39.276142 24539 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 702) took 3892 ms. Trace:
I20260925 03:04:39.273191 26107 raft_consensus.cc:482] T edcf6e6a422d4d198d4fbb0f7d3d845d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:39.303130 25165 leader_election.cc:302] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:04:39.303541 24470 rpcz_store.cc:276] 0925 03:04:35.447763 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:39.287900 (+3840137us) service_pool.cc:227] Handling call
0925 03:04:39.288184 (+ 284us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:39.288339 (+ 155us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:39.288362 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:39.288400 (+ 38us) raft_consensus.cc:1631] Marking committed up to 5
0925 03:04:39.288427 (+ 27us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:39.288475 (+ 48us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:39.288563 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:39.308074 25979 consensus_queue.cc:1225] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer 869b0a8575b54d08ae2817fad5dd42d5 log is divergent from this leader: its last log entry 3.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260925 03:04:39.305902 24586 leader_election.cc:302] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:39.296370 25897 raft_consensus.cc:482] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:39.338037 25165 leader_election.cc:302] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5; no voters:
I20260925 03:04:39.312052 24471 raft_consensus.cc:2454] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 5.
W20260925 03:04:39.294467 24585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.4:43979, user_credentials={real_user=slave}} blocked reactor thread for 625870us
I20260925 03:04:39.347807 24437 leader_election.cc:302] T 21c43af5e6844fdaacc8c625fa01fd8d P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
W20260925 03:04:39.303375 26247 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.597s user 0.053s sys 0.099s
I20260925 03:04:39.315109 26199 raft_consensus.cc:504] T 8e5819c15eac4544a57e5ee6c9d4f117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:39.244822 26291 raft_consensus.cc:482] T 11bb46bce3564841b7674e6a693bd2e6 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:39.320307 26244 leader_election.cc:288] T 367277daf8d04778b3fa4ed94d71580d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:39.297991 26138 raft_consensus.cc:2726] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260925 03:04:39.251487 26262 raft_consensus.cc:2781] T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Leader election won for term 5
I20260925 03:04:39.301566 26181 leader_election.cc:288] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:39.349665 24437 leader_election.cc:302] T 7993f9c3e2a44f408bbf5d1e395e2306 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:04:39.304535 26290 raft_consensus.cc:2781] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260925 03:04:39.340289 24472 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b030d84c670b48d5bff337ccc0c77626"
dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
mode: GRACEFUL
new_leader_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
from {username='slave'} at 127.0.0.1:38832
I20260925 03:04:39.351603 25165 leader_election.cc:302] T b1e85248d3b04b979dd080caeeb02fdb P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5; no voters:
I20260925 03:04:39.349232 26077 raft_consensus.cc:2781] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260925 03:04:39.332320 26196 raft_consensus.cc:3037] T edcf6e6a422d4d198d4fbb0f7d3d845d P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:39.250705 26255 consensus_queue.cc:237] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:39.304261 26068 consensus_queue.cc:1035] T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.043s
I20260925 03:04:39.358791 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ac285175464fbda67211d9574db415" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:39.360188 24437 leader_election.cc:302] T 0be4e0aa984f4f2b81a58e382e434ad5 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:04:39.360325 24615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abd5b8dc1b7e49e3aa9af900d73ff0e7" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:39.350492 26247 raft_consensus.cc:686] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [term 5 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:39.363010 26290 raft_consensus.cc:482] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:39.348876 26107 raft_consensus.cc:3037] T edcf6e6a422d4d198d4fbb0f7d3d845d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:39.353852 26292 raft_consensus.cc:2781] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260925 03:04:39.390861 24618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48ca6ccd98194cf2af7b5278d15bf4ea" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:39.355172 25979 raft_consensus.cc:2781] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader election won for term 4
I20260925 03:04:39.336974 26254 raft_consensus.cc:2781] T 420163a6cdce412ebc21c25fe3835638 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260925 03:04:39.369665 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceab6b0fccd54e909a64f66fe81a9a1f" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:39.145076 26267 consensus_queue.cc:1046] T 442d4a29b1d743599f4b6b4871e26668 P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.228s
I20260925 03:04:39.391412 24465 raft_consensus.cc:2379] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1bc1f2e62ab44af995165bd5116a9d43 in current term 4: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:04:39.423579 24468 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 756) took 3977 ms. Trace:
I20260925 03:04:39.423591 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35992316cb8d4c028333388eff02ecde" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
W20260925 03:04:39.428162 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 70476.8us
I20260925 03:04:39.429343 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21c43af5e6844fdaacc8c625fa01fd8d" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:39.464527 24615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e92f377a79354a6ebb4eec67303c13e3" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:39.363435 26077 raft_consensus.cc:482] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:39.482384 24617 raft_consensus.cc:2454] T 21c43af5e6844fdaacc8c625fa01fd8d P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 1.
I20260925 03:04:39.465939 24618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f68804eb5a244d87bb395d50683c6012" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:39.431964 24511 leader_election.cc:302] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
W20260925 03:04:39.372406 25942 consensus_meta.cc:287] T 71f461a6e2ff434993b78a2d077f0abd P eeab93abfdb141a895002ec9f312296c: Time spent flushing consensus metadata: real 0.627s user 0.007s sys 0.000s
W20260925 03:04:39.472359 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 138880us
I20260925 03:04:39.426995 24616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:39.468091 26292 raft_consensus.cc:482] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:39.367460 26284 raft_consensus.cc:2781] T 21c43af5e6844fdaacc8c625fa01fd8d P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260925 03:04:39.424721 26279 raft_consensus.cc:2781] T 7993f9c3e2a44f408bbf5d1e395e2306 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260925 03:04:39.355716 25897 raft_consensus.cc:504] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:39.371686 25190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:39.372367 25193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e92f377a79354a6ebb4eec67303c13e3" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:39.370779 26262 raft_consensus.cc:686] T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c [term 5 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:39.357512 26175 consensus_queue.cc:1035] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.075s
I20260925 03:04:39.486383 24514 leader_election.cc:302] T c85f5d1d3f004cf5817bb594fd8e6469 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:39.533084 26244 raft_consensus.cc:2726] T c85f5d1d3f004cf5817bb594fd8e6469 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260925 03:04:39.481653 24468 rpcz_store.cc:276] 0925 03:04:35.446665 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:38.742810 (+3296145us) service_pool.cc:227] Handling call
0925 03:04:38.784998 (+ 42188us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:38.785184 (+ 186us) raft_consensus.cc:1520] Early marking committed up to index 2
0925 03:04:39.070982 (+285798us) log.cc:842] Serialized 14 byte log entry
0925 03:04:39.419528 (+348546us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:39.423246 (+ 3718us) raft_consensus.cc:1631] Marking committed up to 2
0925 03:04:39.423293 (+ 47us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:39.423359 (+ 66us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:39.423525 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":93179,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":38,"raft.queue_time_us":130,"raft.run_cpu_time_us":152,"raft.run_wall_time_us":7201,"thread_start_us":585303,"threads_started":2}
I20260925 03:04:39.533699 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43366a0a2f9d4344bf4e37ffd290ef8a" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
W20260925 03:04:39.533934 26272 log.cc:924] Time spent T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.198s user 0.000s sys 0.000s
I20260925 03:04:39.534132 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7aa47c48404465b94a36fd819fe3c7a" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:39.534325 24619 raft_consensus.cc:2454] T 43366a0a2f9d4344bf4e37ffd290ef8a P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 1.
I20260925 03:04:39.534741 24468 raft_consensus.cc:2454] T a7aa47c48404465b94a36fd819fe3c7a P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 2.
I20260925 03:04:39.535161 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367277daf8d04778b3fa4ed94d71580d" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:39.535614 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37a157996e944db2906110d87ecb0c54" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:39.536262 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b5df303cd6a4d59aca5c18bbbff24ed" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:39.483577 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9c33fd100ff44c1a3247319a324a044" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:39.537926 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "420163a6cdce412ebc21c25fe3835638" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:39.538497 24617 raft_consensus.cc:2454] T 420163a6cdce412ebc21c25fe3835638 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 4.
I20260925 03:04:39.483495 24615 raft_consensus.cc:2396] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260925 03:04:39.369545 26150 raft_consensus.cc:2781] T b1e85248d3b04b979dd080caeeb02fdb P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260925 03:04:39.484848 24542 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 713) took 4097 ms. Trace:
I20260925 03:04:39.478508 26254 raft_consensus.cc:482] T 420163a6cdce412ebc21c25fe3835638 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
W20260925 03:04:39.490918 24511 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 59016.9us
I20260925 03:04:39.485525 24466 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 747) took 5088 ms. Trace:
I20260925 03:04:39.471555 26290 raft_consensus.cc:3037] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:39.525707 24618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e068a0f80941dc9138cee063f53bab" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:39.524523 24613 raft_consensus.cc:2454] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 4.
W20260925 03:04:39.550845 26296 log.cc:924] Time spent T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.181s user 0.000s sys 0.001s
W20260925 03:04:39.551467 26289 log.cc:924] Time spent T 3d89396576884f7c8901dba54f783947 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.084s user 0.000s sys 0.000s
I20260925 03:04:39.508478 25942 raft_consensus.cc:504] T 71f461a6e2ff434993b78a2d077f0abd P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
W20260925 03:04:39.472849 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 112735us
I20260925 03:04:39.552299 24437 leader_election.cc:302] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:04:39.552597 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b627ecb4a5384ef392aa695e06599c39" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:39.482414 24586 leader_election.cc:302] T 5278a60f97da4e10aecd50a8d9a9a43c P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:39.536578 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1e85248d3b04b979dd080caeeb02fdb" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:39.520547 24616 raft_consensus.cc:2454] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 4.
W20260925 03:04:39.533231 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 46955.9us
I20260925 03:04:39.539335 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1414dcfe53f64a78ae413531eb8ec117" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:39.544999 24615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1e85248d3b04b979dd080caeeb02fdb" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
W20260925 03:04:39.555609 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 73287.2us
I20260925 03:04:39.556244 24615 raft_consensus.cc:2379] T b1e85248d3b04b979dd080caeeb02fdb P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 in current term 3: Already voted for candidate eeab93abfdb141a895002ec9f312296c in this term.
I20260925 03:04:39.482975 25979 raft_consensus.cc:686] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [term 4 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:39.547339 24588 leader_election.cc:302] T a7aa47c48404465b94a36fd819fe3c7a P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:39.547933 24466 rpcz_store.cc:276] 0925 03:04:34.397196 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:38.627462 (+4230266us) service_pool.cc:227] Handling call
0925 03:04:38.653299 (+ 25837us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:04:38.659448 (+ 6149us) raft_consensus.cc:1520] Early marking committed up to index 2
0925 03:04:38.718877 (+ 59429us) log.cc:842] Serialized 14 byte log entry
0925 03:04:39.093955 (+375078us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:04:39.111531 (+ 17576us) log.cc:842] Serialized 26 byte log entry
0925 03:04:39.111807 (+ 276us) raft_consensus.cc:1631] Marking committed up to 2
0925 03:04:39.111872 (+ 65us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:39.112019 (+ 147us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:04:39.485330 (+373311us) raft_consensus.cc:1671] finished
0925 03:04:39.485351 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:39.485473 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":124948,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":53,"raft.queue_time_us":385852,"raft.run_cpu_time_us":163,"raft.run_wall_time_us":155,"thread_start_us":375407,"threads_started":3,"wal-append.queue_time_us":114112}
I20260925 03:04:39.474457 26277 raft_consensus.cc:2781] T 0be4e0aa984f4f2b81a58e382e434ad5 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260925 03:04:39.532341 25190 raft_consensus.cc:2454] T 29637f5a4a504ce6ba29033ccd0cd716 P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 3.
I20260925 03:04:39.551318 24618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7993f9c3e2a44f408bbf5d1e395e2306" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:39.488220 26077 raft_consensus.cc:3037] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43 [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:04:39.553404 24468 raft_consensus.cc:3037] T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:39.553072 24544 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 720) took 3978 ms. Trace:
I20260925 03:04:39.520251 26292 raft_consensus.cc:3037] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:04:39.554553 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:39.530774 26284 raft_consensus.cc:482] T 21c43af5e6844fdaacc8c625fa01fd8d P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:39.555565 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9c33fd100ff44c1a3247319a324a044" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:39.544540 24542 rpcz_store.cc:276] 0925 03:04:35.387279 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:38.612634 (+3225355us) service_pool.cc:227] Handling call
0925 03:04:38.636625 (+ 23991us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:39.089808 (+453183us) raft_consensus.cc:1520] Early marking committed up to index 3
0925 03:04:39.250331 (+160523us) log.cc:842] Serialized 14 byte log entry
0925 03:04:39.464778 (+214447us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:39.464831 (+ 53us) raft_consensus.cc:1631] Marking committed up to 3
0925 03:04:39.464872 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:39.484566 (+ 19694us) spinlock_profiling.cc:243] Waited 453 ms on lock 0x7b540010cf54. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d0f51 00007f43764cf99e 00007f437b4112e9 00007f4374af7684 00007f4374af7612 00007f4374af7541 00007f4374af74bc 00007f4374af6772 00007f43748fa94c 00007f43748f9d86
0925 03:04:39.484601 (+ 35us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:39.484727 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":48353,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":34,"raft.queue_time_us":843,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":6205,"spinlock_wait_cycles":1041693056,"thread_start_us":349719,"threads_started":2,"wal-append.queue_time_us":239360}
I20260925 03:04:39.556816 24586 leader_election.cc:302] T ac9357d144aa49789faaf31fe5e6bf89 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:04:39.526834 26138 raft_consensus.cc:2741] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260925 03:04:39.559894 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442d4a29b1d743599f4b6b4871e26668" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:39.549782 24511 leader_election.cc:302] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 1bc1f2e62ab44af995165bd5116a9d43, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:39.529721 26279 raft_consensus.cc:482] T 7993f9c3e2a44f408bbf5d1e395e2306 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:39.546459 26150 raft_consensus.cc:482] T b1e85248d3b04b979dd080caeeb02fdb P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260925 03:04:39.553499 26294 log.cc:924] Time spent T abd5b8dc1b7e49e3aa9af900d73ff0e7 P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.268s user 0.000s sys 0.001s
I20260925 03:04:39.275085 24541 rpcz_store.cc:276] 0925 03:04:35.824364 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:38.692138 (+2867774us) service_pool.cc:227] Handling call
0925 03:04:38.741886 (+ 49748us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:38.781053 (+ 39167us) raft_consensus.cc:1520] Early marking committed up to index 2
0925 03:04:38.938764 (+157711us) log.cc:842] Serialized 14 byte log entry
0925 03:04:39.072413 (+133649us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:39.076845 (+ 4432us) raft_consensus.cc:1631] Marking committed up to 2
0925 03:04:39.082337 (+ 5492us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:39.097274 (+ 14937us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:39.126701 (+ 29427us) inbound_call.cc:173] Queueing success response
0925 03:04:39.138696 (+ 11995us) spinlock_profiling.cc:243] Waited 4.35 ms on lock 0x7b48000c1ed8. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f437b3777f1 00007f437b36e8e7 00007f437b36e77a 00007f437b46295f 00007f437b462627 00007f437b476b2c 00007f437b476ad8 00007f437b476a58 00007f437b476a13 00007f437b475cbc 00007f437a3f7be6
Metrics: {"mutex_wait_us":50415,"prepare.queue_time_us":349747,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":34,"raft.queue_time_us":58311,"raft.run_cpu_time_us":2218,"raft.run_wall_time_us":68921,"spinlock_wait_cycles":10003712,"thread_start_us":106166,"threads_started":2,"wal-append.queue_time_us":302681}
I20260925 03:04:39.554618 25957 raft_consensus.cc:504] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:39.602389 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60d45b7d467d4e1ab92367f6d7d52b15" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:39.603263 24547 raft_consensus.cc:2379] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c in current term 5: Already voted for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in this term.
I20260925 03:04:39.603343 24473 raft_consensus.cc:2379] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c in current term 6: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:04:39.603585 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b030d84c670b48d5bff337ccc0c77626" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:39.604158 24542 raft_consensus.cc:2454] T b030d84c670b48d5bff337ccc0c77626 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 4.
I20260925 03:04:39.605073 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:39.605381 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37a157996e944db2906110d87ecb0c54" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:39.569133 24615 raft_consensus.cc:1226] T 37a157996e944db2906110d87ecb0c54 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 4. Current term is 5. Ops: []
W20260925 03:04:39.570158 24514 leader_election.cc:334] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187): Remote error: Timed out: Call waited in the queue past client deadline
I20260925 03:04:39.609845 24473 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 786) took 1151 ms. Trace:
I20260925 03:04:39.609845 24548 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 1256) took 4006 ms. Trace:
I20260925 03:04:39.553345 26253 raft_consensus.cc:2781] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260925 03:04:39.611737 24620 raft_consensus.cc:1226] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 4. Current term is 5. Ops: []
I20260925 03:04:39.610944 24548 rpcz_store.cc:276] 0925 03:04:35.603219 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:38.659102 (+3055883us) service_pool.cc:227] Handling call
0925 03:04:38.685933 (+ 26831us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:04:38.686153 (+ 220us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:38.686174 (+ 21us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:04:38.686568 (+ 394us) log.cc:842] Serialized 26 byte log entry
0925 03:04:38.748842 (+ 62274us) raft_consensus.cc:1631] Marking committed up to 3
0925 03:04:39.246773 (+497931us) log.cc:842] Serialized 14 byte log entry
0925 03:04:39.246977 (+ 204us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:39.247046 (+ 69us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:04:39.605187 (+358141us) raft_consensus.cc:1671] finished
0925 03:04:39.605203 (+ 16us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:39.609779 (+ 4576us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":388784,"prepare.run_cpu_time_us":24,"prepare.run_wall_time_us":25,"raft.queue_time_us":116,"raft.run_cpu_time_us":128,"raft.run_wall_time_us":9258,"thread_start_us":513375,"threads_started":2,"wal-append.queue_time_us":587763}
I20260925 03:04:39.610867 24514 leader_election.cc:302] T 21c43af5e6844fdaacc8c625fa01fd8d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:04:39.610195 24545 raft_consensus.cc:2454] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 4.
I20260925 03:04:39.570523 24544 rpcz_store.cc:276] 0925 03:04:35.574194 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:38.643187 (+3068993us) service_pool.cc:227] Handling call
0925 03:04:38.643501 (+ 314us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:04:38.643902 (+ 401us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:38.643928 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:04:38.644319 (+ 391us) log.cc:842] Serialized 26 byte log entry
0925 03:04:39.277640 (+633321us) raft_consensus.cc:1631] Marking committed up to 2
0925 03:04:39.303276 (+ 25636us) log.cc:842] Serialized 14 byte log entry
0925 03:04:39.304788 (+ 1512us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:39.304861 (+ 73us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:04:39.552890 (+248029us) raft_consensus.cc:1671] finished
0925 03:04:39.552910 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:39.553020 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":26816,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":36,"raft.queue_time_us":1561,"raft.run_cpu_time_us":166,"raft.run_wall_time_us":6986,"spinlock_wait_cycles":6405504,"thread_start_us":607120,"threads_started":1,"wal-append.queue_time_us":704921}
I20260925 03:04:39.576486 24618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c88cbd39e01f44628a094af516be2833" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:39.551494 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be4e0aa984f4f2b81a58e382e434ad5" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:39.568878 26277 raft_consensus.cc:482] T 0be4e0aa984f4f2b81a58e382e434ad5 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:39.585047 26284 raft_consensus.cc:3037] T 21c43af5e6844fdaacc8c625fa01fd8d P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:39.574281 26269 raft_consensus.cc:3032] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [term 4 LEADER]: Stepping down as leader of term 4
W20260925 03:04:39.579084 26120 consensus_meta.cc:287] T 74443a0a1dc34d88a2f42e3b722db405 P eeab93abfdb141a895002ec9f312296c: Time spent flushing consensus metadata: real 1.202s user 0.010s sys 0.001s
I20260925 03:04:39.587060 24511 leader_election.cc:302] T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 1bc1f2e62ab44af995165bd5116a9d43, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:39.587651 26138 raft_consensus.cc:2726] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260925 03:04:39.590880 25897 leader_election.cc:288] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:39.591171 24465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e54ca1757ee4e42956a9cebbeca4f62" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:39.594409 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcf6e6a422d4d198d4fbb0f7d3d845d" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:39.553596 24469 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 761) took 4101 ms. Trace:
I20260925 03:04:39.549472 26254 raft_consensus.cc:3037] T 420163a6cdce412ebc21c25fe3835638 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:39.611896 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e92f377a79354a6ebb4eec67303c13e3" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:39.582873 26305 raft_consensus.cc:2781] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260925 03:04:39.613922 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f22ddd7d1c444dc58f45a1fb0b3e02bc" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:39.671983 24543 raft_consensus.cc:3037] T f22ddd7d1c444dc58f45a1fb0b3e02bc P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:39.609851 24546 raft_consensus.cc:3037] T 37a157996e944db2906110d87ecb0c54 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:39.560531 26068 raft_consensus.cc:2781] T 5278a60f97da4e10aecd50a8d9a9a43c P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260925 03:04:39.616616 24586 leader_election.cc:302] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:39.574755 24616 raft_consensus.cc:1226] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260925 03:04:39.586000 25938 raft_consensus.cc:504] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:04:39.622649 24358 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.GetConsensusState -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 520014us
I20260925 03:04:39.593994 26279 raft_consensus.cc:3037] T 7993f9c3e2a44f408bbf5d1e395e2306 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:39.624337 24468 raft_consensus.cc:2454] T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 3.
I20260925 03:04:39.608258 26150 raft_consensus.cc:3037] T b1e85248d3b04b979dd080caeeb02fdb P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:39.626622 24618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5f9b503b0ab43999d582015a367441e" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:39.627547 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "100c113a9fd84cc8adf2ec7b33ea0093" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:39.692735 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7aa47c48404465b94a36fd819fe3c7a" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:39.628340 24511 leader_election.cc:302] T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 1bc1f2e62ab44af995165bd5116a9d43, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:39.625541 26269 raft_consensus.cc:729] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [term 4 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:39.613925 24473 rpcz_store.cc:276] 0925 03:04:38.458645 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:38.752151 (+293506us) service_pool.cc:227] Handling call
0925 03:04:39.605136 (+852985us) spinlock_profiling.cc:243] Waited 824 ms on lock 0x7b540019f854. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d42ae 00007f437b411c27 00007f4374af9c04 00007f4374af9b92 00007f4374af9ac1 00007f4374af9a3c 00007f4374af8cf2 00007f43748fa94c 00007f43748f9d86 00007f43748f972f
0925 03:04:39.609785 (+ 4649us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1894989568}
W20260925 03:04:39.694625 24511 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 66397.4us
I20260925 03:04:39.695413 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e5819c15eac4544a57e5ee6c9d4f117" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:39.695969 24473 raft_consensus.cc:2362] T 8e5819c15eac4544a57e5ee6c9d4f117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 3. Re-sending same reply.
I20260925 03:04:39.696223 26256 raft_consensus.cc:2726] T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260925 03:04:39.696779 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43366a0a2f9d4344bf4e37ffd290ef8a" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:39.697518 24473 raft_consensus.cc:2379] T 43366a0a2f9d4344bf4e37ffd290ef8a P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c in current term 2: Already voted for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in this term.
I20260925 03:04:39.698063 25194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e54ca1757ee4e42956a9cebbeca4f62" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43"
I20260925 03:04:39.698236 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74443a0a1dc34d88a2f42e3b722db405" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:39.699208 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f1fac028a144058b92331c0d9b44ebf" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:39.699764 24473 raft_consensus.cc:2454] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 4.
I20260925 03:04:39.629887 24514 leader_election.cc:302] T 43366a0a2f9d4344bf4e37ffd290ef8a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:04:39.612798 26267 raft_consensus.cc:2781] T ac9357d144aa49789faaf31fe5e6bf89 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Leader election won for term 2
I20260925 03:04:39.641940 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "420163a6cdce412ebc21c25fe3835638" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:39.701170 26240 raft_consensus.cc:2781] T 43366a0a2f9d4344bf4e37ffd290ef8a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 03:04:39.630566 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43366a0a2f9d4344bf4e37ffd290ef8a" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:39.490437 26293 consensus_queue.cc:1046] T 442d4a29b1d743599f4b6b4871e26668 P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.132s
I20260925 03:04:39.701882 26240 raft_consensus.cc:686] T 43366a0a2f9d4344bf4e37ffd290ef8a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 LEADER]: Becoming Leader. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:04:39.600874 26275 consensus_queue.cc:1035] T 7993f9c3e2a44f408bbf5d1e395e2306 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.025s
I20260925 03:04:39.702893 26240 consensus_queue.cc:237] T 43366a0a2f9d4344bf4e37ffd290ef8a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:39.622012 26277 raft_consensus.cc:3037] T 0be4e0aa984f4f2b81a58e382e434ad5 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Advancing to term 4
W20260925 03:04:39.674368 26303 log.cc:924] Time spent T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.118s user 0.000s sys 0.000s
I20260925 03:04:39.693499 25957 leader_election.cc:288] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 election: Requested vote from peers eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:39.575632 26247 consensus_queue.cc:237] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:39.694708 26269 consensus_queue.cc:260] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:39.707875 24548 raft_consensus.cc:3037] T 100c113a9fd84cc8adf2ec7b33ea0093 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Advancing to term 3
W20260925 03:04:39.709487 26248 consensus_meta.cc:287] T c85f5d1d3f004cf5817bb594fd8e6469 P eeab93abfdb141a895002ec9f312296c: Time spent flushing consensus metadata: real 0.642s user 0.008s sys 0.002s
I20260925 03:04:39.705327 26269 raft_consensus.cc:3037] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:39.682497 26068 raft_consensus.cc:482] T 5278a60f97da4e10aecd50a8d9a9a43c P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:39.713487 26068 raft_consensus.cc:3037] T 5278a60f97da4e10aecd50a8d9a9a43c P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:39.700510 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60d45b7d467d4e1ab92367f6d7d52b15" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:39.720525 24543 raft_consensus.cc:2454] T f22ddd7d1c444dc58f45a1fb0b3e02bc P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 5.
I20260925 03:04:39.674916 26107 raft_consensus.cc:504] T edcf6e6a422d4d198d4fbb0f7d3d845d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:39.633145 26086 raft_consensus.cc:2781] T 21c43af5e6844fdaacc8c625fa01fd8d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Leader election won for term 2
W20260925 03:04:39.675619 26241 consensus_meta.cc:287] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent flushing consensus metadata: real 0.533s user 0.010s sys 0.000s
I20260925 03:04:39.532408 26262 consensus_queue.cc:237] T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:39.671788 26305 raft_consensus.cc:482] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:39.646935 24469 rpcz_store.cc:276] 0925 03:04:35.448361 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:39.300079 (+3851718us) service_pool.cc:227] Handling call
0925 03:04:39.308159 (+ 8080us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:39.308364 (+ 205us) raft_consensus.cc:1520] Early marking committed up to index 3
0925 03:04:39.308647 (+ 283us) log.cc:842] Serialized 14 byte log entry
0925 03:04:39.488404 (+179757us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:39.490282 (+ 1878us) raft_consensus.cc:1631] Marking committed up to 3
0925 03:04:39.520028 (+ 29746us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:39.539129 (+ 19101us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:39.549481 (+ 10352us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":4260,"prepare.run_cpu_time_us":1484,"prepare.run_wall_time_us":18400,"raft.queue_time_us":99960,"raft.run_cpu_time_us":95,"raft.run_wall_time_us":97,"thread_start_us":163570,"threads_started":1,"wal-append.queue_time_us":277149}
I20260925 03:04:39.741488 26107 leader_election.cc:288] T edcf6e6a422d4d198d4fbb0f7d3d845d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:39.742362 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21c43af5e6844fdaacc8c625fa01fd8d" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:39.742702 25938 leader_election.cc:288] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:39.630620 26120 raft_consensus.cc:504] T 74443a0a1dc34d88a2f42e3b722db405 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:39.630486 26239 raft_consensus.cc:504] T 8ac466793d094325ace224708b2158ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:39.627207 26199 leader_election.cc:288] T 8e5819c15eac4544a57e5ee6c9d4f117 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
W20260925 03:04:39.674400 26084 consensus_meta.cc:287] T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c: Time spent flushing consensus metadata: real 1.201s user 0.011s sys 0.000s
W20260925 03:04:39.684406 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 67847.1us
I20260925 03:04:39.630034 25804 raft_consensus.cc:2726] T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260925 03:04:39.690177 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f22ddd7d1c444dc58f45a1fb0b3e02bc" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:39.691630 26077 raft_consensus.cc:504] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:39.701122 26267 raft_consensus.cc:686] T ac9357d144aa49789faaf31fe5e6bf89 P eeab93abfdb141a895002ec9f312296c [term 2 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
W20260925 03:04:39.701289 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 71484.1us
I20260925 03:04:39.777787 24514 leader_election.cc:398] T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 election: Vote denied by peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187) with higher term. Message: Invalid argument: T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 4. Current term is 5.
I20260925 03:04:39.778129 24514 leader_election.cc:401] T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20260925 03:04:39.670195 24615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c23337a7cb74184a0e57b7095cb6a11" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:39.668931 24541 raft_consensus.cc:1261] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260925 03:04:39.611071 26271 raft_consensus.cc:504] T cc2df40db08649c9a75e90000b878559 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:39.735612 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f22ddd7d1c444dc58f45a1fb0b3e02bc" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:39.731376 24618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e8af2641e124f9589c5ccfdac785f63" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:39.737732 24616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367277daf8d04778b3fa4ed94d71580d" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:39.738252 26241 raft_consensus.cc:504] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:39.741331 24547 raft_consensus.cc:1261] T a7aa47c48404465b94a36fd819fe3c7a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260925 03:04:39.793900 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "293ef69caf514ac98510a68495f54eda" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:39.357942 26291 raft_consensus.cc:504] T 11bb46bce3564841b7674e6a693bd2e6 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:39.746971 24546 raft_consensus.cc:2454] T 37a157996e944db2906110d87ecb0c54 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 5.
I20260925 03:04:39.795610 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:39.796387 24546 raft_consensus.cc:2379] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1bc1f2e62ab44af995165bd5116a9d43 in current term 4: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:04:39.747823 24588 leader_election.cc:302] T b1e85248d3b04b979dd080caeeb02fdb P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters: 1bc1f2e62ab44af995165bd5116a9d43
I20260925 03:04:39.748508 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9357d144aa49789faaf31fe5e6bf89" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:39.750396 24548 raft_consensus.cc:2454] T 100c113a9fd84cc8adf2ec7b33ea0093 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 3.
I20260925 03:04:39.769225 24622 raft_consensus.cc:2379] T 74443a0a1dc34d88a2f42e3b722db405 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 5: Already voted for candidate eeab93abfdb141a895002ec9f312296c in this term.
I20260925 03:04:39.773505 24468 raft_consensus.cc:3037] T f22ddd7d1c444dc58f45a1fb0b3e02bc P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Advancing to term 5
W20260925 03:04:39.758023 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 205791us
I20260925 03:04:39.798266 24437 leader_election.cc:302] T b627ecb4a5384ef392aa695e06599c39 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
W20260925 03:04:39.797506 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 49764.4us
I20260925 03:04:39.768862 26084 raft_consensus.cc:2726] T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260925 03:04:39.801126 24437 leader_election.cc:302] T 293ef69caf514ac98510a68495f54eda P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: 1bc1f2e62ab44af995165bd5116a9d43, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:39.801236 24588 leader_election.cc:302] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:39.717264 26270 raft_consensus.cc:2781] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260925 03:04:39.813876 24437 leader_election.cc:302] T e6b6ec010caf4689b8c90449764f5cf9 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 1bc1f2e62ab44af995165bd5116a9d43
I20260925 03:04:39.827899 24618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5aec38e66a947c681e6c2a72b05e6ca" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:39.828935 24437 leader_election.cc:302] T 442d4a29b1d743599f4b6b4871e26668 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:04:39.829954 24620 raft_consensus.cc:1226] T cc2df40db08649c9a75e90000b878559 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260925 03:04:39.709888 26248 raft_consensus.cc:504] T c85f5d1d3f004cf5817bb594fd8e6469 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:39.852116 24618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:39.859650 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b8e0d260a884ecdb20f79de46c36b19" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:39.860299 24466 raft_consensus.cc:2362] T 7b8e0d260a884ecdb20f79de46c36b19 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 4. Re-sending same reply.
W20260925 03:04:39.743556 26209 consensus_meta.cc:287] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5: Time spent flushing consensus metadata: real 0.658s user 0.009s sys 0.000s
I20260925 03:04:39.757781 24471 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 765) took 4308 ms. Trace:
I20260925 03:04:39.779361 24615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcf6e6a422d4d198d4fbb0f7d3d845d" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:39.752805 24465 raft_consensus.cc:2379] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c in current term 6: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:04:39.781514 24542 raft_consensus.cc:2454] T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 3.
I20260925 03:04:39.747457 26305 raft_consensus.cc:3037] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:39.797580 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15861cd28e354598a3c6bfc4337c34bb" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:39.805574 24468 raft_consensus.cc:2454] T f22ddd7d1c444dc58f45a1fb0b3e02bc P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 5.
I20260925 03:04:39.875326 24548 raft_consensus.cc:2454] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 4.
I20260925 03:04:39.672533 25979 consensus_queue.cc:237] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:39.768930 26086 raft_consensus.cc:686] T 21c43af5e6844fdaacc8c625fa01fd8d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 LEADER]: Becoming Leader. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:04:39.852546 24437 leader_election.cc:302] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:04:39.863538 24619 raft_consensus.cc:2454] T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 4.
I20260925 03:04:39.864245 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:39.753965 26284 raft_consensus.cc:504] T 21c43af5e6844fdaacc8c625fa01fd8d P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:39.865844 24471 rpcz_store.cc:276] 0925 03:04:35.449570 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:39.347522 (+3897952us) service_pool.cc:227] Handling call
0925 03:04:39.347838 (+ 316us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:39.348336 (+ 498us) raft_consensus.cc:1520] Early marking committed up to index 2
0925 03:04:39.377756 (+ 29420us) log.cc:842] Serialized 14 byte log entry
0925 03:04:39.757424 (+379668us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:39.757479 (+ 55us) raft_consensus.cc:1631] Marking committed up to 2
0925 03:04:39.757524 (+ 45us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:39.757592 (+ 68us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:39.757717 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3873,"prepare.queue_time_us":29238,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":39,"raft.queue_time_us":63649,"raft.run_cpu_time_us":53,"raft.run_wall_time_us":53,"thread_start_us":324237,"threads_started":2,"wal-append.queue_time_us":151599}
I20260925 03:04:39.336551 24539 rpcz_store.cc:276] 0925 03:04:35.380473 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:38.595139 (+3214666us) service_pool.cc:227] Handling call
0925 03:04:38.595607 (+ 468us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:04:38.596257 (+ 650us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:38.596289 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:04:38.596799 (+ 510us) log.cc:842] Serialized 26 byte log entry
0925 03:04:38.695298 (+ 98499us) raft_consensus.cc:1631] Marking committed up to 1
0925 03:04:38.705843 (+ 10545us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:38.713272 (+ 7429us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:04:39.251606 (+538334us) raft_consensus.cc:1671] finished
0925 03:04:39.251682 (+ 76us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:39.272757 (+ 21075us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":108,"raft.run_cpu_time_us":91,"raft.run_wall_time_us":92,"thread_start_us":92941,"threads_started":1,"wal-append.queue_time_us":148233}
I20260925 03:04:39.868767 24618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ac285175464fbda67211d9574db415" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:39.806793 26285 raft_consensus.cc:2781] T b1e85248d3b04b979dd080caeeb02fdb P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader election won for term 3
I20260925 03:04:39.891062 26120 leader_election.cc:288] T 74443a0a1dc34d88a2f42e3b722db405 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:39.873049 24615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1e85248d3b04b979dd080caeeb02fdb" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:39.877034 24588 leader_election.cc:302] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:39.827852 26275 raft_consensus.cc:2781] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260925 03:04:39.879170 24617 raft_consensus.cc:1226] T c85f5d1d3f004cf5817bb594fd8e6469 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 3. Current term is 4. Ops: []
I20260925 03:04:39.897848 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "293ef69caf514ac98510a68495f54eda" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:39.808118 26138 raft_consensus.cc:2726] T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187) with higher term. Message: Invalid argument: T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 4. Current term is 5.
I20260925 03:04:39.881764 24546 raft_consensus.cc:2379] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1bc1f2e62ab44af995165bd5116a9d43 in current term 4: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:04:39.864995 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15861cd28e354598a3c6bfc4337c34bb" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:39.882520 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "100c113a9fd84cc8adf2ec7b33ea0093" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:39.901533 24588 leader_election.cc:302] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:04:39.902068 24466 raft_consensus.cc:3037] T 100c113a9fd84cc8adf2ec7b33ea0093 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:39.883095 24622 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38564 (request call id 1238) took 1201 ms. Trace:
I20260925 03:04:39.902521 24622 rpcz_store.cc:276] 0925 03:04:38.681324 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:38.684118 (+ 2794us) service_pool.cc:227] Handling call
0925 03:04:39.882909 (+1198791us) spinlock_profiling.cc:243] Waited 1.07 s on lock 0x7b54003232d4. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d42ae 00007f437b411c27 00007f4374af9c04 00007f4374af9b92 00007f4374af9ac1 00007f4374af9a3c 00007f4374af8cf2 00007f43748fa94c 00007f43748f9d86 00007f43748f972f
0925 03:04:39.883036 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2452024576}
I20260925 03:04:39.883565 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "550118bc1a0641ce9803bdbf43f2069f" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:39.834513 26199 raft_consensus.cc:2726] T 293ef69caf514ac98510a68495f54eda P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260925 03:04:39.851845 25938 raft_consensus.cc:2781] T e6b6ec010caf4689b8c90449764f5cf9 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260925 03:04:39.852361 25957 raft_consensus.cc:2781] T 442d4a29b1d743599f4b6b4871e26668 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader election won for term 5
I20260925 03:04:39.831727 26310 raft_consensus.cc:2781] T b627ecb4a5384ef392aa695e06599c39 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260925 03:04:39.834610 25942 leader_election.cc:288] T 71f461a6e2ff434993b78a2d077f0abd P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:39.851405 26270 raft_consensus.cc:482] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:39.888469 24437 leader_election.cc:398] T c88cbd39e01f44628a094af516be2833 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 election: Vote denied by peer a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389) with higher term. Message: Invalid argument: T c88cbd39e01f44628a094af516be2833 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 4. Current term is 5.
I20260925 03:04:39.892266 26077 leader_election.cc:288] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 6 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
W20260925 03:04:39.892787 26307 log.cc:924] Time spent T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.118s user 0.001s sys 0.000s
I20260925 03:04:39.893208 24621 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38564 (request call id 1239) took 1201 ms. Trace:
I20260925 03:04:39.918521 24621 rpcz_store.cc:276] 0925 03:04:38.691607 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:38.691753 (+ 146us) service_pool.cc:227] Handling call
0925 03:04:39.893034 (+1201281us) spinlock_profiling.cc:243] Waited 1.16 s on lock 0x7b54002b0f54. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d42ae 00007f437b411c27 00007f4374af9c04 00007f4374af9b92 00007f4374af9ac1 00007f4374af9a3c 00007f4374af8cf2 00007f43748fa94c 00007f43748f9d86 00007f43748f972f
0925 03:04:39.893150 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2659309952}
I20260925 03:04:39.893776 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "420163a6cdce412ebc21c25fe3835638" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:39.919392 24585 leader_election.cc:302] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:39.919679 24471 raft_consensus.cc:2454] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 4.
I20260925 03:04:39.870308 26209 raft_consensus.cc:504] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:39.895205 24618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f22ddd7d1c444dc58f45a1fb0b3e02bc" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:39.871830 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c85f5d1d3f004cf5817bb594fd8e6469" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:39.902002 24547 raft_consensus.cc:2454] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 4.
I20260925 03:04:39.904067 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74443a0a1dc34d88a2f42e3b722db405" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:39.859207 25897 consensus_queue.cc:1035] T cc2df40db08649c9a75e90000b878559 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.030s
I20260925 03:04:39.914116 24437 leader_election.cc:401] T c88cbd39e01f44628a094af516be2833 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20260925 03:04:39.894297 26311 raft_consensus.cc:2781] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260925 03:04:39.921867 24588 leader_election.cc:302] T 420163a6cdce412ebc21c25fe3835638 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:39.900893 26275 raft_consensus.cc:482] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting leader election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:39.927346 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ac466793d094325ace224708b2158ed" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:39.897337 24616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e54ca1757ee4e42956a9cebbeca4f62" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:39.916175 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7691c6b475428c82d6181d693d26f3" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:39.897728 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4abe0ea757e4886be4945c5001e529a" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:39.945436 24621 raft_consensus.cc:3037] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:39.904856 26247 raft_consensus.cc:2781] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Leader election won for term 5
I20260925 03:04:39.913102 26120 raft_consensus.cc:3037] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:04:39.917732 26270 raft_consensus.cc:3037] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:04:39.950430 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9c33fd100ff44c1a3247319a324a044" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:39.959836 24471 raft_consensus.cc:2362] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 5. Re-sending same reply.
I20260925 03:04:39.907976 25938 raft_consensus.cc:482] T e6b6ec010caf4689b8c90449764f5cf9 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 1bc1f2e62ab44af995165bd5116a9d43)
I20260925 03:04:39.965783 24472 raft_consensus.cc:595] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 LEADER]: Received request to transfer leadership to TS a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:39.955109 24466 raft_consensus.cc:2454] T 100c113a9fd84cc8adf2ec7b33ea0093 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 3.
I20260925 03:04:39.967124 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc2df40db08649c9a75e90000b878559" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:39.943856 26311 raft_consensus.cc:482] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:39.956588 26291 leader_election.cc:288] T 11bb46bce3564841b7674e6a693bd2e6 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:39.933894 26286 raft_consensus.cc:2781] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260925 03:04:39.956681 26196 raft_consensus.cc:504] T edcf6e6a422d4d198d4fbb0f7d3d845d P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:39.958531 25897 raft_consensus.cc:3037] T c88cbd39e01f44628a094af516be2833 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:39.914709 26310 raft_consensus.cc:482] T b627ecb4a5384ef392aa695e06599c39 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:39.917490 26138 raft_consensus.cc:482] T 7993f9c3e2a44f408bbf5d1e395e2306 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:39.956490 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e5819c15eac4544a57e5ee6c9d4f117" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:39.960783 24467 raft_consensus.cc:1226] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 1. Current term is 3. Ops: []
I20260925 03:04:39.963646 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c88cbd39e01f44628a094af516be2833" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:39.930066 25957 raft_consensus.cc:686] T 442d4a29b1d743599f4b6b4871e26668 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 LEADER]: Becoming Leader. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:04:39.926832 26285 raft_consensus.cc:686] T b1e85248d3b04b979dd080caeeb02fdb P eeab93abfdb141a895002ec9f312296c [term 3 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:39.947554 24469 raft_consensus.cc:2354] T 21c43af5e6844fdaacc8c625fa01fd8d P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c for earlier term 2. Current term is 3.
I20260925 03:04:39.970319 24621 raft_consensus.cc:2454] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 4.
I20260925 03:04:39.947553 26275 raft_consensus.cc:3037] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:39.969434 25938 raft_consensus.cc:3037] T e6b6ec010caf4689b8c90449764f5cf9 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:39.946624 26083 raft_consensus.cc:2781] T 420163a6cdce412ebc21c25fe3835638 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260925 03:04:39.907536 26115 consensus_queue.cc:1035] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.054s
I20260925 03:04:39.980000 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5278a60f97da4e10aecd50a8d9a9a43c" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:39.947818 24465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5332a5d329493991dff994e70ea92c" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:40.030525 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5aec38e66a947c681e6c2a72b05e6ca" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:39.977813 26310 raft_consensus.cc:3037] T b627ecb4a5384ef392aa695e06599c39 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:39.969807 26239 leader_election.cc:288] T 8ac466793d094325ace224708b2158ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 election: Requested vote from peers eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:39.964138 25194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71f461a6e2ff434993b78a2d077f0abd" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43"
W20260925 03:04:40.057593 26313 log.cc:924] Time spent T c88cbd39e01f44628a094af516be2833 P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.061s user 0.000s sys 0.000s
I20260925 03:04:39.973918 26286 raft_consensus.cc:482] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:40.058676 26286 raft_consensus.cc:3037] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:40.060403 26279 raft_consensus.cc:504] T 7993f9c3e2a44f408bbf5d1e395e2306 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:40.062028 26284 leader_election.cc:288] T 21c43af5e6844fdaacc8c625fa01fd8d P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:40.033265 25190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ac466793d094325ace224708b2158ed" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43"
I20260925 03:04:40.063148 26279 leader_election.cc:288] T 7993f9c3e2a44f408bbf5d1e395e2306 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:40.063772 25957 consensus_queue.cc:237] T 442d4a29b1d743599f4b6b4871e26668 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:39.965121 26247 raft_consensus.cc:686] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c [term 5 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:39.969885 26311 raft_consensus.cc:3037] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:39.897879 26267 consensus_queue.cc:237] T ac9357d144aa49789faaf31fe5e6bf89 P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:04:39.995997 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 107663us
I20260925 03:04:40.041345 26083 raft_consensus.cc:482] T 420163a6cdce412ebc21c25fe3835638 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260925 03:04:40.045444 25163 leader_election.cc:334] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389): Timed out: RequestConsensusVote RPC to 127.22.193.2:36389 timed out after 1.914s (SENT)
I20260925 03:04:40.033785 26241 leader_election.cc:288] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:40.046063 24613 raft_consensus.cc:1226] T edcf6e6a422d4d198d4fbb0f7d3d845d P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 1. Current term is 2. Ops: []
W20260925 03:04:40.046255 26317 log.cc:924] Time spent T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.074s user 0.000s sys 0.000s
I20260925 03:04:40.047075 26254 raft_consensus.cc:504] T 420163a6cdce412ebc21c25fe3835638 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:40.047786 24465 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 768) took 4597 ms. Trace:
I20260925 03:04:40.080381 24465 rpcz_store.cc:276] 0925 03:04:35.450239 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:40.046956 (+4596717us) service_pool.cc:227] Handling call
0925 03:04:40.047301 (+ 345us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:40.047473 (+ 172us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:40.047500 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:40.047540 (+ 40us) raft_consensus.cc:1631] Marking committed up to 3
0925 03:04:40.047573 (+ 33us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:40.047630 (+ 57us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:40.047733 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260925 03:04:40.056798 26316 log.cc:924] Time spent T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.060s user 0.001s sys 0.000s
I20260925 03:04:40.057298 26209 leader_election.cc:288] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:39.951004 26086 consensus_queue.cc:237] T 21c43af5e6844fdaacc8c625fa01fd8d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:40.086267 24467 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55560 (request call id 1184) took 5059 ms. Trace:
I20260925 03:04:40.086752 24467 rpcz_store.cc:276] 0925 03:04:35.026859 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:38.668042 (+3641183us) service_pool.cc:227] Handling call
0925 03:04:38.696252 (+ 28210us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:39.982305 (+1286053us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:40.085992 (+103687us) spinlock_profiling.cc:243] Waited 1.26 s on lock 0x7b5400119ed4. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d0f51 00007f43764cf99e 00007f437b4112e9 00007f4374af7684 00007f4374af7612 00007f4374af7541 00007f4374af74bc 00007f4374af6772 00007f43748fa94c 00007f43748f9d86
0925 03:04:40.086196 (+ 204us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2907707776}
I20260925 03:04:40.088339 24467 raft_consensus.cc:1261] T ceab6b0fccd54e909a64f66fe81a9a1f P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260925 03:04:40.070721 24437 leader_election.cc:302] T a7aa47c48404465b94a36fd819fe3c7a P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:04:40.089668 24467 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 775) took 4353 ms. Trace:
I20260925 03:04:40.090013 24467 rpcz_store.cc:276] 0925 03:04:35.736583 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:40.087623 (+4351040us) service_pool.cc:227] Handling call
0925 03:04:40.087990 (+ 367us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:40.088664 (+ 674us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:40.089613 (+ 949us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:40.066198 26286 raft_consensus.cc:504] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:40.105077 24511 leader_election.cc:302] T 8ac466793d094325ace224708b2158ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters: 1bc1f2e62ab44af995165bd5116a9d43
I20260925 03:04:40.041857 24616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f1fac028a144058b92331c0d9b44ebf" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:39.983893 26138 raft_consensus.cc:504] T 7993f9c3e2a44f408bbf5d1e395e2306 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:40.042735 26271 leader_election.cc:288] T cc2df40db08649c9a75e90000b878559 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:40.107848 24588 leader_election.cc:302] T 100c113a9fd84cc8adf2ec7b33ea0093 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
W20260925 03:04:40.158517 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 46365.2us
I20260925 03:04:40.169059 24616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35992316cb8d4c028333388eff02ecde" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:40.169833 24616 raft_consensus.cc:3037] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:40.170745 24545 raft_consensus.cc:2454] T 420163a6cdce412ebc21c25fe3835638 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 4.
I20260925 03:04:40.074904 26083 raft_consensus.cc:3037] T 420163a6cdce412ebc21c25fe3835638 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:40.103958 24541 raft_consensus.cc:1226] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Rejecting Update request from peer 1bc1f2e62ab44af995165bd5116a9d43 for earlier term 3. Current term is 4. Ops: []
W20260925 03:04:40.172986 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 65234.3us
I20260925 03:04:40.077231 26248 leader_election.cc:288] T c85f5d1d3f004cf5817bb594fd8e6469 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:40.173766 24586 leader_election.cc:302] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:40.097903 26264 raft_consensus.cc:3032] T 442d4a29b1d743599f4b6b4871e26668 P eeab93abfdb141a895002ec9f312296c [term 4 LEADER]: Stepping down as leader of term 4
I20260925 03:04:40.198279 24586 leader_election.cc:302] T 43366a0a2f9d4344bf4e37ffd290ef8a P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:40.124876 26209 raft_consensus.cc:2781] T a7aa47c48404465b94a36fd819fe3c7a P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260925 03:04:40.174932 26292 consensus_meta.cc:287] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5: Time spent flushing consensus metadata: real 0.601s user 0.003s sys 0.008s
I20260925 03:04:40.206868 24436 leader_election.cc:302] T ceab6b0fccd54e909a64f66fe81a9a1f P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:40.122401 26241 raft_consensus.cc:2781] T 8ac466793d094325ace224708b2158ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader election won for term 4
I20260925 03:04:40.215621 24436 leader_election.cc:302] T 43366a0a2f9d4344bf4e37ffd290ef8a P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:40.215863 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71f461a6e2ff434993b78a2d077f0abd" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:40.211774 24586 leader_election.cc:302] T 37a157996e944db2906110d87ecb0c54 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5
W20260925 03:04:40.176391 26257 consensus_meta.cc:287] T 25102ba08e4e49388f6176badcd10a25 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent flushing consensus metadata: real 0.903s user 0.011s sys 0.003s
I20260925 03:04:40.045478 26285 consensus_queue.cc:237] T b1e85248d3b04b979dd080caeeb02fdb P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:40.216765 24543 raft_consensus.cc:2379] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c in current term 4: Already voted for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in this term.
I20260925 03:04:40.172942 26077 raft_consensus.cc:482] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:40.187597 25942 raft_consensus.cc:2781] T 100c113a9fd84cc8adf2ec7b33ea0093 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader election won for term 3
I20260925 03:04:40.110749 26196 leader_election.cc:288] T edcf6e6a422d4d198d4fbb0f7d3d845d P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:40.221159 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367277daf8d04778b3fa4ed94d71580d" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:40.228673 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35992316cb8d4c028333388eff02ecde" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:40.197585 26120 raft_consensus.cc:2726] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260925 03:04:40.199537 26264 raft_consensus.cc:729] T 442d4a29b1d743599f4b6b4871e26668 P eeab93abfdb141a895002ec9f312296c [term 4 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:40.205616 26255 raft_consensus.cc:482] T 25102ba08e4e49388f6176badcd10a25 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:40.215567 26209 raft_consensus.cc:482] T a7aa47c48404465b94a36fd819fe3c7a P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:40.214937 26301 raft_consensus.cc:3037] T 43366a0a2f9d4344bf4e37ffd290ef8a P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:40.211735 26248 raft_consensus.cc:2726] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260925 03:04:40.174906 26279 consensus_queue.cc:1035] T 37a157996e944db2906110d87ecb0c54 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.049s
I20260925 03:04:40.253206 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5aec38e66a947c681e6c2a72b05e6ca" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:40.197751 26199 consensus_queue.cc:1035] T 550118bc1a0641ce9803bdbf43f2069f P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.037s
I20260925 03:04:40.204888 26115 consensus_queue.cc:1035] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.032s
I20260925 03:04:40.230705 26241 raft_consensus.cc:686] T 8ac466793d094325ace224708b2158ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 LEADER]: Becoming Leader. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:04:40.235359 26293 raft_consensus.cc:2781] T 37a157996e944db2906110d87ecb0c54 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Leader election won for term 5
I20260925 03:04:40.230512 26077 raft_consensus.cc:504] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:40.269707 26138 leader_election.cc:288] T 7993f9c3e2a44f408bbf5d1e395e2306 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:40.271217 25938 raft_consensus.cc:504] T e6b6ec010caf4689b8c90449764f5cf9 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
W20260925 03:04:40.273195 26321 log.cc:924] Time spent T a4abe0ea757e4886be4945c5001e529a P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.225s user 0.003s sys 0.000s
I20260925 03:04:40.273010 26264 consensus_queue.cc:260] T 442d4a29b1d743599f4b6b4871e26668 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:40.234198 26245 consensus_queue.cc:1035] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.040s
I20260925 03:04:40.295732 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7993f9c3e2a44f408bbf5d1e395e2306" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:40.274451 26254 leader_election.cc:288] T 420163a6cdce412ebc21c25fe3835638 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:40.277886 24616 raft_consensus.cc:2454] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 5.
I20260925 03:04:40.283632 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5278a60f97da4e10aecd50a8d9a9a43c" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:40.281066 26253 raft_consensus.cc:482] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 FOLLOWER]: Starting leader election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:40.297981 24619 raft_consensus.cc:1226] T ac9357d144aa49789faaf31fe5e6bf89 P eeab93abfdb141a895002ec9f312296c [term 2 LEADER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 1. Current term is 2. Ops: []
I20260925 03:04:40.256186 26209 raft_consensus.cc:3037] T a7aa47c48404465b94a36fd819fe3c7a P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:40.208011 26277 raft_consensus.cc:504] T 0be4e0aa984f4f2b81a58e382e434ad5 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:40.288482 24614 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38564 (request call id 1240) took 1596 ms. Trace:
I20260925 03:04:40.237814 26326 raft_consensus.cc:2726] T ceab6b0fccd54e909a64f66fe81a9a1f P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260925 03:04:40.225026 25979 consensus_queue.cc:1035] T a7aa47c48404465b94a36fd819fe3c7a P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.042s
I20260925 03:04:40.224086 26257 raft_consensus.cc:504] T 25102ba08e4e49388f6176badcd10a25 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:40.317817 26277 leader_election.cc:288] T 0be4e0aa984f4f2b81a58e382e434ad5 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:40.318099 26257 leader_election.cc:288] T 25102ba08e4e49388f6176badcd10a25 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:40.223726 26292 raft_consensus.cc:504] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:40.319221 25938 leader_election.cc:288] T e6b6ec010caf4689b8c90449764f5cf9 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:40.319820 26292 leader_election.cc:288] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 election: Requested vote from peers eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:40.262542 25942 raft_consensus.cc:686] T 100c113a9fd84cc8adf2ec7b33ea0093 P eeab93abfdb141a895002ec9f312296c [term 3 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:40.291229 26293 raft_consensus.cc:686] T 37a157996e944db2906110d87ecb0c54 P eeab93abfdb141a895002ec9f312296c [term 5 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:40.320971 25942 consensus_queue.cc:237] T 100c113a9fd84cc8adf2ec7b33ea0093 P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:40.292905 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21c43af5e6844fdaacc8c625fa01fd8d" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:40.274421 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bb46bce3564841b7674e6a693bd2e6" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:40.241525 26325 raft_consensus.cc:2781] T 43366a0a2f9d4344bf4e37ffd290ef8a P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260925 03:04:40.210433 26247 consensus_queue.cc:237] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:40.299410 24616 raft_consensus.cc:1226] T 74443a0a1dc34d88a2f42e3b722db405 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 4. Current term is 5. Ops: []
I20260925 03:04:40.321455 26293 consensus_queue.cc:237] T 37a157996e944db2906110d87ecb0c54 P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:40.323608 26311 raft_consensus.cc:504] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:40.301440 26253 raft_consensus.cc:3037] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:40.301702 26275 raft_consensus.cc:504] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:40.302906 26310 raft_consensus.cc:504] T b627ecb4a5384ef392aa695e06599c39 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:40.329896 26310 leader_election.cc:288] T b627ecb4a5384ef392aa695e06599c39 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
W20260925 03:04:40.300567 26150 consensus_meta.cc:287] T b1e85248d3b04b979dd080caeeb02fdb P 1bc1f2e62ab44af995165bd5116a9d43: Time spent flushing consensus metadata: real 0.595s user 0.004s sys 0.007s
I20260925 03:04:40.330626 26150 raft_consensus.cc:504] T b1e85248d3b04b979dd080caeeb02fdb P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:40.298170 25190 raft_consensus.cc:1226] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 5. Current term is 6. Ops: [5.6-5.6]
W20260925 03:04:40.301522 26320 log.cc:924] Time spent T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.271s user 0.001s sys 0.000s
I20260925 03:04:40.324625 26325 raft_consensus.cc:482] T 43366a0a2f9d4344bf4e37ffd290ef8a P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:40.336598 26325 raft_consensus.cc:3037] T 43366a0a2f9d4344bf4e37ffd290ef8a P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:40.337860 26311 leader_election.cc:288] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
W20260925 03:04:40.307289 26269 consensus_meta.cc:287] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c: Time spent flushing consensus metadata: real 0.597s user 0.006s sys 0.003s
W20260925 03:04:40.299372 26314 log.cc:924] Time spent T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.258s user 0.000s sys 0.001s
I20260925 03:04:40.309932 26305 raft_consensus.cc:504] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:04:40.350827 26290 consensus_meta.cc:287] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5: Time spent flushing consensus metadata: real 0.792s user 0.008s sys 0.002s
I20260925 03:04:40.350942 26305 leader_election.cc:288] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:40.310472 26301 raft_consensus.cc:2726] T 43366a0a2f9d4344bf4e37ffd290ef8a P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260925 03:04:40.301227 26322 log.cc:924] Time spent T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5: Append to log took a long time: real 0.122s user 0.000s sys 0.001s
W20260925 03:04:40.308714 26068 consensus_meta.cc:287] T 5278a60f97da4e10aecd50a8d9a9a43c P eeab93abfdb141a895002ec9f312296c: Time spent flushing consensus metadata: real 0.595s user 0.009s sys 0.001s
I20260925 03:04:40.332628 26150 leader_election.cc:288] T b1e85248d3b04b979dd080caeeb02fdb P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 4 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:40.352998 26068 raft_consensus.cc:504] T 5278a60f97da4e10aecd50a8d9a9a43c P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:40.355154 26068 leader_election.cc:288] T 5278a60f97da4e10aecd50a8d9a9a43c P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 3 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:40.345811 26077 leader_election.cc:288] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:40.346271 26241 consensus_queue.cc:237] T 8ac466793d094325ace224708b2158ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:40.348384 26275 leader_election.cc:288] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:40.348811 26264 raft_consensus.cc:3037] T 442d4a29b1d743599f4b6b4871e26668 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:40.363775 26083 raft_consensus.cc:504] T 420163a6cdce412ebc21c25fe3835638 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:40.403769 24464 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 794) took 1702 ms. Trace:
W20260925 03:04:40.403648 24436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 187893us
I20260925 03:04:40.412921 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c85f5d1d3f004cf5817bb594fd8e6469" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:40.414321 24466 raft_consensus.cc:2379] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 1bc1f2e62ab44af995165bd5116a9d43 in current term 6: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:04:40.283422 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1414dcfe53f64a78ae413531eb8ec117" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:40.493737 24436 leader_election.cc:302] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:40.495712 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcf6e6a422d4d198d4fbb0f7d3d845d" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:40.496973 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:40.498098 24544 raft_consensus.cc:2454] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 4.
I20260925 03:04:40.324188 24616 raft_consensus.cc:1226] T 71f461a6e2ff434993b78a2d077f0abd P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Rejecting Update request from peer 1bc1f2e62ab44af995165bd5116a9d43 for earlier term 3. Current term is 4. Ops: []
I20260925 03:04:40.270375 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "420163a6cdce412ebc21c25fe3835638" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:40.327109 24617 raft_consensus.cc:1226] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Rejecting Update request from peer 1bc1f2e62ab44af995165bd5116a9d43 for earlier term 5. Current term is 6. Ops: []
I20260925 03:04:40.501729 24545 raft_consensus.cc:2454] T 420163a6cdce412ebc21c25fe3835638 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 4.
I20260925 03:04:40.502096 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74443a0a1dc34d88a2f42e3b722db405" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:40.417371 24620 raft_consensus.cc:2379] T 5278a60f97da4e10aecd50a8d9a9a43c P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 in current term 3: Already voted for candidate eeab93abfdb141a895002ec9f312296c in this term.
I20260925 03:04:40.327498 24622 raft_consensus.cc:3037] T 21c43af5e6844fdaacc8c625fa01fd8d P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Advancing to term 3
I20260925 03:04:40.499519 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abd5b8dc1b7e49e3aa9af900d73ff0e7" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:40.256112 26255 raft_consensus.cc:504] T 25102ba08e4e49388f6176badcd10a25 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:40.504976 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcf6e6a422d4d198d4fbb0f7d3d845d" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:40.372334 26286 leader_election.cc:288] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:40.377692 26290 raft_consensus.cc:504] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:40.414685 24471 raft_consensus.cc:1226] T 0be4e0aa984f4f2b81a58e382e434ad5 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 3. Current term is 4. Ops: []
I20260925 03:04:40.503041 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bb46bce3564841b7674e6a693bd2e6" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:40.266095 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43366a0a2f9d4344bf4e37ffd290ef8a" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:40.312446 26270 raft_consensus.cc:504] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:40.507787 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c85f5d1d3f004cf5817bb594fd8e6469" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:40.316540 24614 rpcz_store.cc:276] 0925 03:04:38.691827 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:38.691925 (+ 98us) service_pool.cc:227] Handling call
0925 03:04:40.283318 (+1591393us) spinlock_profiling.cc:243] Waited 1.28 s on lock 0x7b54000a6254. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d42ae 00007f437b411c27 00007f4374af9c04 00007f4374af9b92 00007f4374af9ac1 00007f4374af9a3c 00007f4374af8cf2 00007f43748fa94c 00007f43748f9d86 00007f43748f972f
0925 03:04:40.288409 (+ 5091us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2941424384}
I20260925 03:04:40.459383 24619 raft_consensus.cc:1226] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Rejecting Update request from peer 1bc1f2e62ab44af995165bd5116a9d43 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260925 03:04:40.425180 25897 raft_consensus.cc:2726] T c88cbd39e01f44628a094af516be2833 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389) with higher term. Message: Invalid argument: T c88cbd39e01f44628a094af516be2833 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 4. Current term is 5.
I20260925 03:04:40.509737 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:40.461510 24467 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 783) took 2097 ms. Trace:
I20260925 03:04:40.511010 24467 rpcz_store.cc:276] 0925 03:04:38.363800 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:40.107568 (+1743768us) service_pool.cc:227] Handling call
0925 03:04:40.168488 (+ 60920us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:40.168712 (+ 224us) raft_consensus.cc:1520] Early marking committed up to index 2
0925 03:04:40.254777 (+ 86065us) log.cc:842] Serialized 14 byte log entry
0925 03:04:40.460538 (+205761us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:40.460665 (+ 127us) raft_consensus.cc:1631] Marking committed up to 2
0925 03:04:40.460885 (+ 220us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:40.461038 (+ 153us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:40.461275 (+ 237us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":65736,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":39,"raft.queue_time_us":570,"raft.run_cpu_time_us":251,"raft.run_wall_time_us":252,"thread_start_us":273246,"threads_started":2,"wal-append.queue_time_us":254553}
I20260925 03:04:40.427682 26325 raft_consensus.cc:504] T 43366a0a2f9d4344bf4e37ffd290ef8a P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:40.513530 24467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "420163a6cdce412ebc21c25fe3835638" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:40.506896 26255 leader_election.cc:288] T 25102ba08e4e49388f6176badcd10a25 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:40.510854 24468 raft_consensus.cc:2379] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c in current term 5: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:04:40.521896 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f1fac028a144058b92331c0d9b44ebf" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:40.522941 24468 raft_consensus.cc:3037] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:40.508361 26290 leader_election.cc:288] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 election: Requested vote from peers eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:40.526240 24470 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 763) took 5077 ms. Trace:
I20260925 03:04:40.526743 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:40.527642 24542 raft_consensus.cc:2362] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 4. Re-sending same reply.
I20260925 03:04:40.528177 26325 leader_election.cc:288] T 43366a0a2f9d4344bf4e37ffd290ef8a P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:40.528851 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5278a60f97da4e10aecd50a8d9a9a43c" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:40.529743 24547 raft_consensus.cc:1261] T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Refusing update from remote peer 1bc1f2e62ab44af995165bd5116a9d43: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 5. (term mismatch)
I20260925 03:04:40.530207 24547 pending_rounds.cc:76] T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Aborting all ops after (but not including) 4
I20260925 03:04:40.530524 24547 pending_rounds.cc:97] T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Aborting uncommitted NO_OP operation due to leader change: 2.5
I20260925 03:04:40.471846 24464 rpcz_store.cc:276] 0925 03:04:38.701273 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:38.736442 (+ 35169us) service_pool.cc:227] Handling call
0925 03:04:40.242394 (+1505952us) spinlock_profiling.cc:243] Waited 1.23 s on lock 0x7b5400191ed4. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d42ae 00007f437b411c27 00007f4374af9c04 00007f4374af9b92 00007f4374af9ac1 00007f4374af9a3c 00007f4374af8cf2 00007f43748fa94c 00007f43748f9d86 00007f43748f972f
0925 03:04:40.403450 (+161056us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2824792960}
I20260925 03:04:40.509372 26270 leader_election.cc:288] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 6 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:40.532155 24464 raft_consensus.cc:1261] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260925 03:04:40.532783 24464 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 787) took 2073 ms. Trace:
I20260925 03:04:40.533183 24464 rpcz_store.cc:276] 0925 03:04:38.458926 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:40.531140 (+2072214us) service_pool.cc:227] Handling call
0925 03:04:40.531697 (+ 557us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:40.532503 (+ 806us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:40.532679 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:40.536757 24472 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:38832 (request call id 8) took 1207 ms. Trace:
I20260925 03:04:40.537138 24472 rpcz_store.cc:276] 0925 03:04:39.329088 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:39.340035 (+ 10947us) service_pool.cc:227] Handling call
0925 03:04:40.536397 (+1196362us) spinlock_profiling.cc:243] Waited 605 ms on lock 0x7b5400191ed4. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764cabc3 00007f437b41481a 00007f4374b07d04 00007f4374b07c92 00007f4374b07bc1 00007f4374b07b3c 00007f4374b06df2 00007f43748fa94c 00007f43748f9d86 00007f43748f972f
0925 03:04:40.536555 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1391095680}
W20260925 03:04:40.488587 25163 leader_election.cc:334] T a4abe0ea757e4886be4945c5001e529a P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389): Timed out: RequestConsensusVote RPC to 127.22.193.2:36389 timed out after 2.078s (SENT)
I20260925 03:04:40.497232 26305 raft_consensus.cc:2781] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260925 03:04:40.538993 26305 raft_consensus.cc:482] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:40.539556 26305 raft_consensus.cc:3037] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:40.538964 25163 leader_election.cc:302] T a4abe0ea757e4886be4945c5001e529a P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:40.462756 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60d45b7d467d4e1ab92367f6d7d52b15" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:40.534718 24464 raft_consensus.cc:1226] T 7993f9c3e2a44f408bbf5d1e395e2306 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 3. Current term is 4. Ops: []
I20260925 03:04:40.530002 24466 raft_consensus.cc:2379] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c in current term 3: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:04:40.532514 24471 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 767) took 5082 ms. Trace:
I20260925 03:04:40.416676 24621 raft_consensus.cc:1226] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 4. Current term is 5. Ops: []
I20260925 03:04:40.539860 24547 raft_consensus.cc:2866] T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260925 03:04:40.503031 24548 raft_consensus.cc:3037] T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:40.539474 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b5df303cd6a4d59aca5c18bbbff24ed" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:40.462193 26083 leader_election.cc:288] T 420163a6cdce412ebc21c25fe3835638 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:40.541079 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b8e0d260a884ecdb20f79de46c36b19" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:40.541446 24471 rpcz_store.cc:276] 0925 03:04:35.450053 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:40.043093 (+4593040us) service_pool.cc:227] Handling call
0925 03:04:40.070687 (+ 27594us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:40.506853 (+436166us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:40.532237 (+ 25384us) spinlock_profiling.cc:243] Waited 343 ms on lock 0x7b5400091354. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d0f51 00007f43764cf99e 00007f437b4112e9 00007f4374af7684 00007f4374af7612 00007f4374af7541 00007f4374af74bc 00007f4374af6772 00007f43748fa94c 00007f43748f9d86
0925 03:04:40.532460 (+ 223us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":789741312}
I20260925 03:04:40.510874 24469 raft_consensus.cc:2379] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c in current term 5: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:04:40.508646 24545 raft_consensus.cc:3037] T c85f5d1d3f004cf5817bb594fd8e6469 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:40.526695 24470 rpcz_store.cc:276] 0925 03:04:35.449012 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:39.338564 (+3889552us) service_pool.cc:227] Handling call
0925 03:04:39.338902 (+ 338us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:39.852534 (+513632us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:39.852566 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:39.852604 (+ 38us) raft_consensus.cc:1631] Marking committed up to 2
0925 03:04:39.852635 (+ 31us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:40.525749 (+673114us) spinlock_profiling.cc:243] Waited 513 ms on lock 0x7b54003106d4. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d0f51 00007f43764cf99e 00007f437b4112e9 00007f4374af7684 00007f4374af7612 00007f4374af7541 00007f4374af74bc 00007f4374af6772 00007f43748fa94c 00007f43748f9d86
0925 03:04:40.525813 (+ 64us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:40.526048 (+ 235us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1180775296}
I20260925 03:04:40.549703 24464 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 792) took 1847 ms. Trace:
I20260925 03:04:40.552538 24464 rpcz_store.cc:276] 0925 03:04:38.700830 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:40.533928 (+1833098us) service_pool.cc:227] Handling call
0925 03:04:40.534367 (+ 439us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:40.545771 (+ 11404us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:40.547658 (+ 1887us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:40.508451 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06f4a7eaf6534e4986c9fd35911038e4" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:40.560429 24542 raft_consensus.cc:2354] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 3. Current term is 4.
I20260925 03:04:40.563196 24540 raft_consensus.cc:2354] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 4. Current term is 5.
W20260925 03:04:40.564257 25163 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 143653us
I20260925 03:04:40.569298 24588 leader_election.cc:398] T 21c43af5e6844fdaacc8c625fa01fd8d P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Vote denied by peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187) with higher term. Message: Invalid argument: T 21c43af5e6844fdaacc8c625fa01fd8d P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c for earlier term 2. Current term is 3.
I20260925 03:04:40.569615 26337 raft_consensus.cc:2726] T a4abe0ea757e4886be4945c5001e529a P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260925 03:04:40.570991 24543 raft_consensus.cc:2454] T 0b5df303cd6a4d59aca5c18bbbff24ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 4.
I20260925 03:04:40.571949 24470 raft_consensus.cc:1226] T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Rejecting Update request from peer 1bc1f2e62ab44af995165bd5116a9d43 for earlier term 2. Current term is 3. Ops: []
I20260925 03:04:40.572986 24588 leader_election.cc:401] T 21c43af5e6844fdaacc8c625fa01fd8d P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260925 03:04:40.573611 24544 raft_consensus.cc:2379] T edcf6e6a422d4d198d4fbb0f7d3d845d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c in current term 2: Already voted for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in this term.
I20260925 03:04:40.578438 24614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1e85248d3b04b979dd080caeeb02fdb" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:40.580068 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ac285175464fbda67211d9574db415" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:40.580830 24622 raft_consensus.cc:2454] T 21c43af5e6844fdaacc8c625fa01fd8d P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 3.
I20260925 03:04:40.581418 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21c43af5e6844fdaacc8c625fa01fd8d" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:40.580271 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c81a1b3b616492ab6c964f45af36077" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:40.585932 24471 raft_consensus.cc:1226] T e6b6ec010caf4689b8c90449764f5cf9 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Rejecting Update request from peer 1bc1f2e62ab44af995165bd5116a9d43 for earlier term 3. Current term is 4. Ops: []
I20260925 03:04:40.586584 24471 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38920 (request call id 82) took 1852 ms. Trace:
I20260925 03:04:40.586995 24547 raft_consensus.cc:3037] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:40.589135 25190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43"
I20260925 03:04:40.589221 24368 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:57656 (request call id 82) took 10041 ms. Trace:
I20260925 03:04:40.594111 24464 raft_consensus.cc:1226] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Rejecting Update request from peer 1bc1f2e62ab44af995165bd5116a9d43 for earlier term 5. Current term is 6. Ops: []
I20260925 03:04:40.590037 24469 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38920 (request call id 83) took 1840 ms. Trace:
I20260925 03:04:40.594732 24464 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38920 (request call id 88) took 1654 ms. Trace:
I20260925 03:04:40.595059 24464 rpcz_store.cc:276] 0925 03:04:38.941192 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:40.563425 (+1622233us) service_pool.cc:227] Handling call
0925 03:04:40.593822 (+ 30397us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:40.594500 (+ 678us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:40.594663 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260925 03:04:40.597442 24436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 101471us
I20260925 03:04:40.590911 24368 rpcz_store.cc:276] 0925 03:04:30.547772 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:30.548181 (+ 409us) service_pool.cc:227] Handling call
0925 03:04:40.588975 (+10040794us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0925 03:04:38.298197 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0925 03:04:38.298709 (+ 512us) write_op.cc:432] Acquiring schema lock in shared mode
0925 03:04:38.298743 (+ 34us) write_op.cc:435] Acquired schema lock
0925 03:04:38.298757 (+ 14us) tablet.cc:660] Decoding operations
0925 03:04:38.299537 (+ 780us) write_op.cc:620] Acquiring the partition lock for write op
0925 03:04:38.299640 (+ 103us) write_op.cc:641] Partition lock acquired for write op
0925 03:04:38.299690 (+ 50us) tablet.cc:683] Acquiring locks for 3 operations
0925 03:04:38.300089 (+ 399us) tablet.cc:699] Row locks acquired
0925 03:04:38.300106 (+ 17us) write_op.cc:260] PREPARE: finished
0925 03:04:38.300238 (+ 132us) write_op.cc:270] Start()
0925 03:04:38.300343 (+ 105us) write_op.cc:276] Timestamp: P: 1790305478300196 usec, L: 0
0925 03:04:38.300365 (+ 22us) op_driver.cc:331] REPLICATION: starting
0925 03:04:38.301513 (+ 1148us) log.cc:842] Serialized 1350 byte log entry
0925 03:04:38.937260 (+635747us) spinlock_profiling.cc:243] Waited 6.3 ms on lock 0x7b1800012d00. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f437c4478b1 00007f437c454661 00007f43735a8638 00007f43735abb11 00007f43735abac9 00007f43735aba59 00007f43735aba21 00007f43735aad1d 00007f437c80c204 00007f437c80c039 00007f437358adc6
0925 03:04:39.525382 (+588122us) op_driver.cc:444] REPLICATION: finished
0925 03:04:39.750052 (+224670us) write_op.cc:301] APPLY: starting
0925 03:04:39.750133 (+ 81us) tablet.cc:1365] starting BulkCheckPresence
0925 03:04:39.750255 (+ 122us) tablet.cc:1368] finished BulkCheckPresence
0925 03:04:39.750274 (+ 19us) tablet.cc:1370] starting ApplyRowOperation cycle
0925 03:04:39.750814 (+ 540us) tablet.cc:1381] finished ApplyRowOperation cycle
0925 03:04:39.750991 (+ 177us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0925 03:04:39.751038 (+ 47us) write_op.cc:312] APPLY: finished
0925 03:04:39.751316 (+ 278us) log.cc:842] Serialized 34 byte log entry
0925 03:04:39.751494 (+ 178us) write_op.cc:489] Releasing partition, row and schema locks
0925 03:04:39.751596 (+ 102us) write_op.cc:454] Released schema lock
0925 03:04:39.751635 (+ 39us) write_op.cc:341] FINISH: Updating metrics
0925 03:04:39.751892 (+ 257us) write_op.cc:489] Releasing partition, row and schema locks
0925 03:04:39.751921 (+ 29us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":16073,"reported_tablets":9,"child_traces":[["op",{"apply.queue_time_us":203159,"apply.run_cpu_time_us":2165,"apply.run_wall_time_us":2231,"num_ops":3,"prepare.queue_time_us":13492,"prepare.run_cpu_time_us":162384,"prepare.run_wall_time_us":219151,"raft.queue_time_us":375305,"raft.run_cpu_time_us":56599,"raft.run_wall_time_us":279327,"replication_time_us":1187991,"spinlock_wait_cycles":14497152,"thread_start_us":682933,"threads_started":4,"wal-append.queue_time_us":221753}]]}
I20260925 03:04:40.599750 24436 leader_election.cc:302] T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: 1bc1f2e62ab44af995165bd5116a9d43, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:40.600761 26325 raft_consensus.cc:2726] T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260925 03:04:40.601480 24541 raft_consensus.cc:2454] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 4.
I20260925 03:04:40.594753 24469 rpcz_store.cc:276] 0925 03:04:38.749759 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:40.586792 (+1837033us) service_pool.cc:227] Handling call
0925 03:04:40.587258 (+ 466us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:40.589683 (+ 2425us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:40.589710 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:40.589761 (+ 51us) raft_consensus.cc:1631] Marking committed up to 5
0925 03:04:40.589786 (+ 25us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:40.589852 (+ 66us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:40.589960 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:40.619065 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "100c113a9fd84cc8adf2ec7b33ea0093" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:40.621142 24469 raft_consensus.cc:2354] T 100c113a9fd84cc8adf2ec7b33ea0093 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 2. Current term is 3.
I20260925 03:04:40.625468 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "550118bc1a0641ce9803bdbf43f2069f" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:40.626314 24546 raft_consensus.cc:2362] T 550118bc1a0641ce9803bdbf43f2069f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 3. Re-sending same reply.
I20260925 03:04:40.586943 24471 rpcz_store.cc:276] 0925 03:04:38.734168 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:40.563084 (+1828916us) service_pool.cc:227] Handling call
0925 03:04:40.585669 (+ 22585us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:40.586349 (+ 680us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:40.586524 (+ 175us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:40.637092 24470 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38920 (request call id 84) took 1800 ms. Trace:
I20260925 03:04:40.642990 24470 rpcz_store.cc:276] 0925 03:04:38.783313 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:40.560759 (+1777446us) service_pool.cc:227] Handling call
0925 03:04:40.571628 (+ 10869us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:40.576613 (+ 4985us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:40.583209 (+ 6596us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:40.625161 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e725b82985746b39c0cc8f671c95353" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:40.590082 25190 raft_consensus.cc:3037] T 29637f5a4a504ce6ba29033ccd0cd716 P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 FOLLOWER]: Advancing to term 5
I20260925 03:04:40.648070 24548 raft_consensus.cc:2454] T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 5.
I20260925 03:04:40.650368 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35992316cb8d4c028333388eff02ecde" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:40.650983 24469 raft_consensus.cc:2379] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 5: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:04:40.620347 26209 raft_consensus.cc:504] T a7aa47c48404465b94a36fd819fe3c7a P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:40.653040 24464 raft_consensus.cc:2454] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 4.
W20260925 03:04:40.655892 25163 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 69822.8us
I20260925 03:04:40.655516 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d89396576884f7c8901dba54f783947" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:40.655563 24547 raft_consensus.cc:2454] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 4.
I20260925 03:04:40.660333 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b030d84c670b48d5bff337ccc0c77626" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:40.662241 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48ca6ccd98194cf2af7b5278d15bf4ea" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:40.660795 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37a157996e944db2906110d87ecb0c54" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:40.662801 24472 raft_consensus.cc:1261] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
W20260925 03:04:40.661103 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 92334.6us
I20260925 03:04:40.664638 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceab6b0fccd54e909a64f66fe81a9a1f" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:40.663945 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b030d84c670b48d5bff337ccc0c77626" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:40.669615 24543 raft_consensus.cc:3037] T b030d84c670b48d5bff337ccc0c77626 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:40.670305 24466 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 802) took 1225 ms. Trace:
I20260925 03:04:40.662565 26270 consensus_queue.cc:1046] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.007s
I20260925 03:04:40.671975 26247 raft_consensus.cc:2726] T 21c43af5e6844fdaacc8c625fa01fd8d P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187) with higher term. Message: Invalid argument: T 21c43af5e6844fdaacc8c625fa01fd8d P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c for earlier term 2. Current term is 3.
I20260925 03:04:40.672761 26247 raft_consensus.cc:3032] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [term 5 LEADER]: Stepping down as leader of term 5
I20260925 03:04:40.673171 26247 raft_consensus.cc:729] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [term 5 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:40.669664 24542 raft_consensus.cc:2454] T 48ca6ccd98194cf2af7b5278d15bf4ea P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 2.
I20260925 03:04:40.674612 24545 raft_consensus.cc:2454] T c85f5d1d3f004cf5817bb594fd8e6469 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 4.
I20260925 03:04:40.674474 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d89396576884f7c8901dba54f783947" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
W20260925 03:04:40.675724 24585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.4:43979, user_credentials={real_user=slave}} blocked reactor thread for 65695.5us
I20260925 03:04:40.676668 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4ebd2b043024a548005c8b6b00b0449" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:40.677614 24472 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 793) took 1974 ms. Trace:
I20260925 03:04:40.677977 24472 rpcz_store.cc:276] 0925 03:04:38.701025 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:40.538116 (+1837091us) service_pool.cc:227] Handling call
0925 03:04:40.655569 (+117453us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:40.663154 (+ 7585us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:40.671507 (+ 8353us) spinlock_profiling.cc:243] Waited 116 ms on lock 0x7b540026fcd0. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764cfa09 00007f437b4112e9 00007f4374af7684 00007f4374af7612 00007f4374af7541 00007f4374af74bc 00007f4374af6772 00007f43748fa94c 00007f43748f9d86 00007f43748f972f
0925 03:04:40.675224 (+ 3717us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":266676608}
I20260925 03:04:40.681118 26337 consensus_queue.cc:1046] T 71f461a6e2ff434993b78a2d077f0abd P 1bc1f2e62ab44af995165bd5116a9d43 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260925 03:04:40.681692 24546 raft_consensus.cc:2396] T 3d89396576884f7c8901dba54f783947 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260925 03:04:40.682466 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15861cd28e354598a3c6bfc4337c34bb" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:40.682650 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5332a5d329493991dff994e70ea92c" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:40.683324 24546 raft_consensus.cc:3037] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:40.671205 24466 rpcz_store.cc:276] 0925 03:04:39.438629 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:40.603284 (+1164655us) service_pool.cc:227] Handling call
0925 03:04:40.603763 (+ 479us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:40.604211 (+ 448us) raft_consensus.cc:1520] Early marking committed up to index 3
0925 03:04:40.604659 (+ 448us) log.cc:842] Serialized 14 byte log entry
0925 03:04:40.663441 (+ 58782us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:40.663479 (+ 38us) raft_consensus.cc:1631] Marking committed up to 3
0925 03:04:40.663507 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:40.663549 (+ 42us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:40.663626 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":914,"prepare.run_cpu_time_us":30,"prepare.run_wall_time_us":31,"raft.queue_time_us":9788,"thread_start_us":49721,"threads_started":1}
I20260925 03:04:40.680608 24473 raft_consensus.cc:2454] T c88cbd39e01f44628a094af516be2833 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 5.
I20260925 03:04:40.684952 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f22ddd7d1c444dc58f45a1fb0b3e02bc" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
W20260925 03:04:40.687139 24436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 87610.2us
I20260925 03:04:40.688782 26210 raft_consensus.cc:3032] T 71f461a6e2ff434993b78a2d077f0abd P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 LEADER]: Stepping down as leader of term 3
I20260925 03:04:40.691847 26210 raft_consensus.cc:729] T 71f461a6e2ff434993b78a2d077f0abd P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1bc1f2e62ab44af995165bd5116a9d43, State: Running, Role: LEADER
I20260925 03:04:40.693045 26210 consensus_queue.cc:260] T 71f461a6e2ff434993b78a2d077f0abd P 1bc1f2e62ab44af995165bd5116a9d43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:40.693897 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21c43af5e6844fdaacc8c625fa01fd8d" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:40.694520 26210 raft_consensus.cc:3037] T 71f461a6e2ff434993b78a2d077f0abd P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:40.695284 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7691c6b475428c82d6181d693d26f3" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:40.696097 24540 raft_consensus.cc:2454] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 5.
I20260925 03:04:40.682080 26209 leader_election.cc:288] T a7aa47c48404465b94a36fd819fe3c7a P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:40.689502 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9357d144aa49789faaf31fe5e6bf89" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:40.690374 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "550118bc1a0641ce9803bdbf43f2069f" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:40.697377 24547 raft_consensus.cc:2379] T f22ddd7d1c444dc58f45a1fb0b3e02bc P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 in current term 5: Already voted for candidate eeab93abfdb141a895002ec9f312296c in this term.
I20260925 03:04:40.698428 24544 raft_consensus.cc:2454] T 550118bc1a0641ce9803bdbf43f2069f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 3.
I20260925 03:04:40.697098 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48ca6ccd98194cf2af7b5278d15bf4ea" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:40.691092 24469 raft_consensus.cc:2454] T ceab6b0fccd54e909a64f66fe81a9a1f P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 4.
I20260925 03:04:40.683806 26305 raft_consensus.cc:504] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:04:40.701357 26345 log.cc:924] Time spent T 37a157996e944db2906110d87ecb0c54 P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.072s user 0.001s sys 0.000s
I20260925 03:04:40.693058 26337 consensus_queue.cc:1046] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 5, Time since last communication: 0.011s
I20260925 03:04:40.704272 26337 raft_consensus.cc:3032] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [term 5 LEADER]: Stepping down as leader of term 5
I20260925 03:04:40.698805 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60d45b7d467d4e1ab92367f6d7d52b15" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:40.704633 26337 raft_consensus.cc:729] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1bc1f2e62ab44af995165bd5116a9d43, State: Running, Role: LEADER
I20260925 03:04:40.708371 24465 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 774) took 4928 ms. Trace:
I20260925 03:04:40.683586 24470 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c85f5d1d3f004cf5817bb594fd8e6469"
dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
mode: GRACEFUL
new_leader_uuid: "eeab93abfdb141a895002ec9f312296c"
from {username='slave'} at 127.0.0.1:38832
I20260925 03:04:40.708858 24465 rpcz_store.cc:276] 0925 03:04:35.736123 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:40.081433 (+4345310us) service_pool.cc:227] Handling call
0925 03:04:40.081795 (+ 362us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:40.081981 (+ 186us) raft_consensus.cc:1520] Early marking committed up to index 4
0925 03:04:40.211153 (+129172us) log.cc:842] Serialized 14 byte log entry
0925 03:04:40.408883 (+197730us) spinlock_profiling.cc:243] Waited 8.19 ms on lock 0x7b1800126e80. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f437c4478b1 00007f437c454661 00007f437358a345 00007f437c3c1a39 00007f43735a78bb 00007f43735a5cbf 00007f43735a7e22 00007f43761515bb 00007f4376157063 00007f437615757b 00007f43764da88a
0925 03:04:40.641267 (+232384us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:40.646856 (+ 5589us) raft_consensus.cc:1631] Marking committed up to 4
0925 03:04:40.648811 (+ 1955us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:40.649756 (+ 945us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:40.664878 (+ 15122us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":115407,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":44,"raft.queue_time_us":11519,"raft.run_cpu_time_us":158,"raft.run_wall_time_us":158,"spinlock_wait_cycles":18841984,"thread_start_us":240150,"threads_started":2,"wal-append.queue_time_us":361884}
I20260925 03:04:40.711211 24465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9c33fd100ff44c1a3247319a324a044" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:40.712184 24465 raft_consensus.cc:2379] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 5: Already voted for candidate eeab93abfdb141a895002ec9f312296c in this term.
I20260925 03:04:40.710482 24470 raft_consensus.cc:595] T c85f5d1d3f004cf5817bb594fd8e6469 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 LEADER]: Received request to transfer leadership to TS eeab93abfdb141a895002ec9f312296c
I20260925 03:04:40.713624 26077 consensus_queue.cc:1046] T 8ac466793d094325ace224708b2158ed P 1bc1f2e62ab44af995165bd5116a9d43 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260925 03:04:40.714262 23300 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 7
I20260925 03:04:40.714618 26077 raft_consensus.cc:3032] T 8ac466793d094325ace224708b2158ed P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 LEADER]: Stepping down as leader of term 3
I20260925 03:04:40.715087 26077 raft_consensus.cc:729] T 8ac466793d094325ace224708b2158ed P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1bc1f2e62ab44af995165bd5116a9d43, State: Running, Role: LEADER
I20260925 03:04:40.718019 23300 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260925 03:04:40.719097 23300 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260925 03:04:40.721969 24588 leader_election.cc:302] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:40.722746 24470 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 812) took 1125 ms. Trace:
I20260925 03:04:40.724658 26248 consensus_queue.cc:1035] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260925 03:04:40.727501 25190 raft_consensus.cc:2454] T 29637f5a4a504ce6ba29033ccd0cd716 P 1bc1f2e62ab44af995165bd5116a9d43 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 5.
I20260925 03:04:40.723028 26246 raft_consensus.cc:2726] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260925 03:04:40.724238 24470 rpcz_store.cc:276] 0925 03:04:39.597605 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:40.720977 (+1123372us) service_pool.cc:227] Handling call
0925 03:04:40.721602 (+ 625us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:40.721934 (+ 332us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:40.721956 (+ 22us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:40.721995 (+ 39us) raft_consensus.cc:1631] Marking committed up to 3
0925 03:04:40.722041 (+ 46us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:40.722574 (+ 533us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:40.722690 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:40.730029 24465 raft_consensus.cc:1226] T b627ecb4a5384ef392aa695e06599c39 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 4. Current term is 5. Ops: []
I20260925 03:04:40.732301 24465 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 813) took 1116 ms. Trace:
I20260925 03:04:40.732851 24465 rpcz_store.cc:276] 0925 03:04:39.616207 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:40.729376 (+1113169us) service_pool.cc:227] Handling call
0925 03:04:40.729776 (+ 400us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:40.732002 (+ 2226us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:40.732249 (+ 247us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:40.737015 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc2df40db08649c9a75e90000b878559" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:40.707619 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5278a60f97da4e10aecd50a8d9a9a43c" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:40.734203 24470 raft_consensus.cc:1226] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 3. Current term is 5. Ops: []
I20260925 03:04:40.737723 24466 raft_consensus.cc:1226] T a7aa47c48404465b94a36fd819fe3c7a P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 2. Current term is 3. Ops: []
I20260925 03:04:40.738911 24466 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 807) took 1293 ms. Trace:
I20260925 03:04:40.739231 24466 rpcz_store.cc:276] 0925 03:04:39.445896 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:40.691538 (+1245642us) service_pool.cc:227] Handling call
0925 03:04:40.737605 (+ 46067us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:40.738746 (+ 1141us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:40.738873 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:40.739416 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f1fac028a144058b92331c0d9b44ebf" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:40.716169 26077 consensus_queue.cc:260] T 8ac466793d094325ace224708b2158ed P 1bc1f2e62ab44af995165bd5116a9d43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:40.741299 24467 raft_consensus.cc:2379] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c in current term 5: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
W20260925 03:04:40.739023 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 419980us
I20260925 03:04:40.742930 24546 raft_consensus.cc:2454] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 5.
I20260925 03:04:40.745893 24543 raft_consensus.cc:2454] T b030d84c670b48d5bff337ccc0c77626 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 5.
I20260925 03:04:40.747071 24548 raft_consensus.cc:3037] T cc2df40db08649c9a75e90000b878559 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:40.749157 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be4e0aa984f4f2b81a58e382e434ad5" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:40.745241 26241 raft_consensus.cc:482] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:40.750458 24544 raft_consensus.cc:3037] T 5278a60f97da4e10aecd50a8d9a9a43c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:40.745283 26150 raft_consensus.cc:482] T 367277daf8d04778b3fa4ed94d71580d P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:40.751277 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f68804eb5a244d87bb395d50683c6012" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:40.752934 24464 raft_consensus.cc:2454] T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 2.
I20260925 03:04:40.754550 24471 raft_consensus.cc:1226] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 5. Current term is 6. Ops: []
I20260925 03:04:40.755576 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:40.756044 24546 raft_consensus.cc:3037] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:40.756532 24543 raft_consensus.cc:3037] T 0be4e0aa984f4f2b81a58e382e434ad5 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:40.756805 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e5819c15eac4544a57e5ee6c9d4f117" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:40.747113 26077 raft_consensus.cc:3037] T 8ac466793d094325ace224708b2158ed P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:40.758810 24540 raft_consensus.cc:2454] T 8e5819c15eac4544a57e5ee6c9d4f117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 3.
I20260925 03:04:40.752286 26150 raft_consensus.cc:504] T 367277daf8d04778b3fa4ed94d71580d P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:40.765393 26241 raft_consensus.cc:504] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:40.766839 26253 raft_consensus.cc:504] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:40.767470 26241 leader_election.cc:288] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:40.767808 26150 leader_election.cc:288] T 367277daf8d04778b3fa4ed94d71580d P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:40.769557 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4abe0ea757e4886be4945c5001e529a" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:40.770756 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7993f9c3e2a44f408bbf5d1e395e2306" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:40.771584 24545 raft_consensus.cc:3037] T 7993f9c3e2a44f408bbf5d1e395e2306 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:40.772657 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b627ecb4a5384ef392aa695e06599c39" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:40.760246 24367 catalog_manager.cc:5495] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P 869b0a8575b54d08ae2817fad5dd42d5 reported cstate change: term changed from 3 to 4, leader changed from a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2) to eeab93abfdb141a895002ec9f312296c (127.22.193.3). New cstate: current_term: 4 leader_uuid: "eeab93abfdb141a895002ec9f312296c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } }
I20260925 03:04:40.774159 24468 raft_consensus.cc:2454] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 5.
I20260925 03:04:40.774884 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6b6ec010caf4689b8c90449764f5cf9" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:40.775874 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:40.776584 24541 raft_consensus.cc:3037] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:40.778534 24473 raft_consensus.cc:1261] T 293ef69caf514ac98510a68495f54eda P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260925 03:04:40.781553 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25102ba08e4e49388f6176badcd10a25" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:40.781944 24618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367277daf8d04778b3fa4ed94d71580d" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:40.782513 24464 raft_consensus.cc:1261] T c88cbd39e01f44628a094af516be2833 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260925 03:04:40.774459 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367277daf8d04778b3fa4ed94d71580d" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:40.786304 24540 raft_consensus.cc:3037] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:40.789654 24542 raft_consensus.cc:2379] T 25102ba08e4e49388f6176badcd10a25 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 in current term 2: Already voted for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in this term.
I20260925 03:04:40.790581 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43366a0a2f9d4344bf4e37ffd290ef8a" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:40.791834 24469 raft_consensus.cc:1226] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 4. Current term is 5. Ops: []
I20260925 03:04:40.737104 26247 consensus_queue.cc:260] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:40.793192 24545 raft_consensus.cc:2454] T 7993f9c3e2a44f408bbf5d1e395e2306 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 4.
I20260925 03:04:40.762261 26337 consensus_queue.cc:260] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:40.794919 24472 raft_consensus.cc:1261] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260925 03:04:40.780511 24367 catalog_manager.cc:5495] T 442d4a29b1d743599f4b6b4871e26668 P 869b0a8575b54d08ae2817fad5dd42d5 reported cstate change: term changed from 3 to 4, leader changed from a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2) to eeab93abfdb141a895002ec9f312296c (127.22.193.3). New cstate: current_term: 4 leader_uuid: "eeab93abfdb141a895002ec9f312296c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } }
I20260925 03:04:40.796494 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35992316cb8d4c028333388eff02ecde" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:40.786798 26253 leader_election.cc:288] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 4 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
W20260925 03:04:40.803514 24436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 61771.1us
I20260925 03:04:40.803731 24542 raft_consensus.cc:2379] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 in current term 5: Already voted for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in this term.
I20260925 03:04:40.800993 24471 raft_consensus.cc:1261] T 8e5819c15eac4544a57e5ee6c9d4f117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260925 03:04:40.799819 24539 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 751) took 1075 ms. Trace:
I20260925 03:04:40.812789 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcf6e6a422d4d198d4fbb0f7d3d845d" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:40.813870 24542 raft_consensus.cc:1226] T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Rejecting Update request from peer 1bc1f2e62ab44af995165bd5116a9d43 for earlier term 2. Current term is 3. Ops: []
I20260925 03:04:40.810720 26209 raft_consensus.cc:482] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:40.818884 24545 raft_consensus.cc:1226] T 8ac466793d094325ace224708b2158ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 LEADER]: Rejecting Update request from peer 1bc1f2e62ab44af995165bd5116a9d43 for earlier term 3. Current term is 4. Ops: []
I20260925 03:04:40.819211 24543 raft_consensus.cc:2454] T 0be4e0aa984f4f2b81a58e382e434ad5 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 4.
I20260925 03:04:40.822566 24542 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33068 (request call id 101) took 1012 ms. Trace:
I20260925 03:04:40.824855 24546 raft_consensus.cc:2454] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 5.
I20260925 03:04:40.815446 26209 raft_consensus.cc:504] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:40.825667 24545 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33068 (request call id 96) took 1030 ms. Trace:
I20260925 03:04:40.822937 24542 rpcz_store.cc:276] 0925 03:04:39.806470 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:40.806530 (+1000060us) service_pool.cc:227] Handling call
0925 03:04:40.809159 (+ 2629us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:40.818199 (+ 9040us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:40.818358 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:40.826157 24545 rpcz_store.cc:276] 0925 03:04:39.790608 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:40.802496 (+1011888us) service_pool.cc:227] Handling call
0925 03:04:40.818761 (+ 16265us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:40.820647 (+ 1886us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:40.820832 (+ 185us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:40.820133 24465 raft_consensus.cc:1226] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 3. Current term is 4. Ops: []
I20260925 03:04:40.832207 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc2df40db08649c9a75e90000b878559" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:40.833905 24548 raft_consensus.cc:2454] T cc2df40db08649c9a75e90000b878559 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 4.
I20260925 03:04:40.818344 24367 catalog_manager.cc:5495] T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5 reported cstate change: term changed from 3 to 4, leader changed from a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2) to 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1). New cstate: current_term: 4 leader_uuid: "869b0a8575b54d08ae2817fad5dd42d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: HEALTHY } } }
I20260925 03:04:40.826752 26305 leader_election.cc:288] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:40.837620 25942 consensus_queue.cc:1046] T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.047s
I20260925 03:04:40.832337 26337 raft_consensus.cc:3037] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:04:40.840813 24543 raft_consensus.cc:1226] T 7993f9c3e2a44f408bbf5d1e395e2306 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260925 03:04:40.835196 24586 leader_election.cc:302] T 71f461a6e2ff434993b78a2d077f0abd P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 1bc1f2e62ab44af995165bd5116a9d43, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:40.841569 24542 raft_consensus.cc:1226] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 4. Current term is 5. Ops: []
I20260925 03:04:40.843132 24548 raft_consensus.cc:1226] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 4. Current term is 5. Ops: []
I20260925 03:04:40.838197 26247 raft_consensus.cc:3037] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:04:40.846447 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7aa47c48404465b94a36fd819fe3c7a" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:40.849241 24470 raft_consensus.cc:2354] T a7aa47c48404465b94a36fd819fe3c7a P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 2. Current term is 3.
I20260925 03:04:40.844072 24465 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 830) took 1047 ms. Trace:
I20260925 03:04:40.854846 24546 raft_consensus.cc:1226] T 37a157996e944db2906110d87ecb0c54 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260925 03:04:40.856204 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e5819c15eac4544a57e5ee6c9d4f117" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:40.852847 26248 raft_consensus.cc:2726] T 71f461a6e2ff434993b78a2d077f0abd P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260925 03:04:40.858052 26209 leader_election.cc:288] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:40.861550 24473 raft_consensus.cc:2354] T 8e5819c15eac4544a57e5ee6c9d4f117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 2. Current term is 3.
W20260925 03:04:40.863511 26352 log.cc:924] Time spent T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.105s user 0.001s sys 0.000s
I20260925 03:04:40.863914 25190 raft_consensus.cc:1226] T b1e85248d3b04b979dd080caeeb02fdb P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 3. Current term is 4. Ops: []
I20260925 03:04:40.866142 25942 consensus_queue.cc:1046] T b1e85248d3b04b979dd080caeeb02fdb P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260925 03:04:40.866909 25942 consensus_queue.cc:1225] T b1e85248d3b04b979dd080caeeb02fdb P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer 1bc1f2e62ab44af995165bd5116a9d43 log is divergent from this leader: its last log entry 2.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260925 03:04:40.867344 26248 raft_consensus.cc:3032] T b1e85248d3b04b979dd080caeeb02fdb P eeab93abfdb141a895002ec9f312296c [term 3 LEADER]: Stepping down as leader of term 3
I20260925 03:04:40.867704 26248 raft_consensus.cc:729] T b1e85248d3b04b979dd080caeeb02fdb P eeab93abfdb141a895002ec9f312296c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:40.852129 24465 rpcz_store.cc:276] 0925 03:04:39.795488 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:40.750774 (+955286us) service_pool.cc:227] Handling call
0925 03:04:40.803529 (+ 52755us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:40.836142 (+ 32613us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:40.842612 (+ 6470us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:40.868525 26248 consensus_queue.cc:260] T b1e85248d3b04b979dd080caeeb02fdb P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
W20260925 03:04:40.875347 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 48712.5us
I20260925 03:04:40.875798 26248 raft_consensus.cc:3037] T b1e85248d3b04b979dd080caeeb02fdb P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:40.882640 24544 raft_consensus.cc:2454] T 5278a60f97da4e10aecd50a8d9a9a43c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 3.
I20260925 03:04:40.886255 25190 raft_consensus.cc:1226] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 3. Current term is 4. Ops: []
W20260925 03:04:40.889405 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 89768.9us
I20260925 03:04:40.890654 26293 consensus_queue.cc:1046] T 7993f9c3e2a44f408bbf5d1e395e2306 P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260925 03:04:40.891570 26247 raft_consensus.cc:3032] T 7993f9c3e2a44f408bbf5d1e395e2306 P eeab93abfdb141a895002ec9f312296c [term 3 LEADER]: Stepping down as leader of term 3
I20260925 03:04:40.893007 24542 raft_consensus.cc:1226] T cc2df40db08649c9a75e90000b878559 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260925 03:04:40.892496 25942 consensus_queue.cc:1035] T 367277daf8d04778b3fa4ed94d71580d P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.002s
I20260925 03:04:40.895453 24543 raft_consensus.cc:1226] T ac9357d144aa49789faaf31fe5e6bf89 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 1. Current term is 2. Ops: []
I20260925 03:04:40.899295 24465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1414dcfe53f64a78ae413531eb8ec117" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
W20260925 03:04:40.899955 24514 connection.cc:232] RPC call timeout handler was delayed by -3.4010750439992989s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1579s for a response
W20260925 03:04:40.900396 24514 connection.cc:232] RPC call timeout handler was delayed by -3.3911750439983734s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15900000000000003s for a response
W20260925 03:04:40.900756 24514 connection.cc:232] RPC call timeout handler was delayed by -3.3785750439983531s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16040000000000001s for a response
W20260925 03:04:40.901038 24514 connection.cc:232] RPC call timeout handler was delayed by -3.1805750439980329s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1824s for a response
W20260925 03:04:40.901381 24514 connection.cc:232] RPC call timeout handler was delayed by -3.0968750439988071s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1917s for a response
W20260925 03:04:40.901803 24514 connection.cc:232] RPC call timeout handler was delayed by -3.0734750439987693s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19430000000000003s for a response
W20260925 03:04:40.902227 24514 connection.cc:232] RPC call timeout handler was delayed by -2.9825750439995318s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20440000000000003s for a response
I20260925 03:04:40.897071 24541 raft_consensus.cc:2454] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 5.
I20260925 03:04:40.902868 24367 catalog_manager.cc:5495] T 06f4a7eaf6534e4986c9fd35911038e4 P 869b0a8575b54d08ae2817fad5dd42d5 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "eeab93abfdb141a895002ec9f312296c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } }
I20260925 03:04:40.905274 24367 catalog_manager.cc:5495] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 reported cstate change: term changed from 3 to 4, leader changed from a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2) to 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1). New cstate: current_term: 4 leader_uuid: "869b0a8575b54d08ae2817fad5dd42d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: UNKNOWN } } }
I20260925 03:04:40.906124 26285 raft_consensus.cc:3032] T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c [term 3 LEADER]: Stepping down as leader of term 3
I20260925 03:04:40.811929 24539 rpcz_store.cc:276] 0925 03:04:39.622425 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:39.902870 (+280445us) service_pool.cc:227] Handling call
0925 03:04:39.935167 (+ 32297us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:04:39.977992 (+ 42825us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:39.982455 (+ 4463us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:04:40.239114 (+256659us) log.cc:842] Serialized 556 byte log entry
0925 03:04:40.657452 (+418338us) raft_consensus.cc:1631] Marking committed up to 4
0925 03:04:40.664446 (+ 6994us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:40.685000 (+ 20554us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:04:40.686167 (+ 1167us) raft_consensus.cc:1671] finished
0925 03:04:40.687594 (+ 1427us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:40.698001 (+ 10407us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":144076,"threads_started":1,"wal-append.queue_time_us":368453}
I20260925 03:04:40.902271 24511 leader_election.cc:302] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:40.892148 26247 raft_consensus.cc:729] T 7993f9c3e2a44f408bbf5d1e395e2306 P eeab93abfdb141a895002ec9f312296c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:40.915284 26241 raft_consensus.cc:2781] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader election won for term 5
I20260925 03:04:40.913456 26285 raft_consensus.cc:729] T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:40.916023 26241 raft_consensus.cc:686] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 LEADER]: Becoming Leader. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:04:40.915522 24586 leader_election.cc:302] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters: 1bc1f2e62ab44af995165bd5116a9d43
I20260925 03:04:40.917282 25942 raft_consensus.cc:2781] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260925 03:04:40.917006 26241 consensus_queue.cc:237] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:40.917274 24465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcf6e6a422d4d198d4fbb0f7d3d845d" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:40.918612 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:40.918969 24436 leader_election.cc:302] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:40.919221 24586 leader_election.cc:302] T edcf6e6a422d4d198d4fbb0f7d3d845d P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:40.918025 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35992316cb8d4c028333388eff02ecde" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:40.919387 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ac285175464fbda67211d9574db415" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:40.920102 24473 raft_consensus.cc:2379] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 5: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:04:40.920208 24469 raft_consensus.cc:2379] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 4: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:04:40.919325 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b8e0d260a884ecdb20f79de46c36b19" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:40.921053 26246 raft_consensus.cc:2726] T edcf6e6a422d4d198d4fbb0f7d3d845d P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260925 03:04:40.922024 24471 raft_consensus.cc:2354] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 4. Current term is 5.
I20260925 03:04:40.917702 25942 raft_consensus.cc:482] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting leader election (detected failure of leader 1bc1f2e62ab44af995165bd5116a9d43)
I20260925 03:04:40.921212 24470 raft_consensus.cc:2354] T 7b8e0d260a884ecdb20f79de46c36b19 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 3. Current term is 4.
I20260925 03:04:40.922677 25942 raft_consensus.cc:3037] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:40.922854 24586 leader_election.cc:302] T 74443a0a1dc34d88a2f42e3b722db405 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5
W20260925 03:04:40.902848 24514 connection.cc:232] RPC call timeout handler was delayed by -2.9636750439985917s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20650000000000002s for a response
W20260925 03:04:40.926589 24514 connection.cc:232] RPC call timeout handler was delayed by -2.8259750439992786s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2218s for a response
W20260925 03:04:40.926800 24514 connection.cc:232] RPC call timeout handler was delayed by -2.57937504399888s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2492s for a response
W20260925 03:04:40.929133 24514 connection.cc:232] RPC call timeout handler was delayed by -2.3579750439985219s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2738s for a response
W20260925 03:04:40.929730 24514 connection.cc:232] RPC call timeout handler was delayed by -2.1293750439981523s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2992s for a response
W20260925 03:04:40.929943 24514 connection.cc:232] RPC call timeout handler was delayed by -1.9610750439987896s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3179s for a response
I20260925 03:04:40.931172 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06f4a7eaf6534e4986c9fd35911038e4" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:40.911149 24367 catalog_manager.cc:5495] T 37a157996e944db2906110d87ecb0c54 P 869b0a8575b54d08ae2817fad5dd42d5 reported cstate change: term changed from 3 to 4, leader changed from a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2) to 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1). New cstate: current_term: 4 leader_uuid: "869b0a8575b54d08ae2817fad5dd42d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: UNKNOWN } } }
I20260925 03:04:40.932308 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c88cbd39e01f44628a094af516be2833" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:40.934446 24471 raft_consensus.cc:2379] T c88cbd39e01f44628a094af516be2833 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 5: Already voted for candidate eeab93abfdb141a895002ec9f312296c in this term.
I20260925 03:04:40.932272 26209 raft_consensus.cc:2726] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260925 03:04:40.937753 24470 raft_consensus.cc:2454] T 06f4a7eaf6534e4986c9fd35911038e4 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 4.
I20260925 03:04:40.937426 24367 catalog_manager.cc:5495] T 9e8af2641e124f9589c5ccfdac785f63 P 869b0a8575b54d08ae2817fad5dd42d5 reported cstate change: term changed from 2 to 3, leader changed from a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2) to eeab93abfdb141a895002ec9f312296c (127.22.193.3). New cstate: current_term: 3 leader_uuid: "eeab93abfdb141a895002ec9f312296c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } }
I20260925 03:04:40.916410 26285 consensus_queue.cc:260] T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:40.940608 26285 raft_consensus.cc:3037] T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:40.916296 26247 consensus_queue.cc:260] T 7993f9c3e2a44f408bbf5d1e395e2306 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:40.941701 26247 raft_consensus.cc:3037] T 7993f9c3e2a44f408bbf5d1e395e2306 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:40.936898 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442d4a29b1d743599f4b6b4871e26668" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:40.946002 26348 consensus_queue.cc:1046] T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260925 03:04:40.946233 26270 consensus_queue.cc:1046] T a7aa47c48404465b94a36fd819fe3c7a P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260925 03:04:40.947232 26270 raft_consensus.cc:3032] T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c [term 4 LEADER]: Stepping down as leader of term 4
I20260925 03:04:40.947636 26246 raft_consensus.cc:3032] T a7aa47c48404465b94a36fd819fe3c7a P eeab93abfdb141a895002ec9f312296c [term 2 LEADER]: Stepping down as leader of term 2
I20260925 03:04:40.950014 26246 raft_consensus.cc:729] T a7aa47c48404465b94a36fd819fe3c7a P eeab93abfdb141a895002ec9f312296c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:40.947633 26270 raft_consensus.cc:729] T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c [term 4 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:40.950412 26209 consensus_queue.cc:1046] T 4e5332a5d329493991dff994e70ea92c P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260925 03:04:40.951453 26246 consensus_queue.cc:260] T a7aa47c48404465b94a36fd819fe3c7a P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:40.952143 26305 raft_consensus.cc:3032] T 4e5332a5d329493991dff994e70ea92c P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 LEADER]: Stepping down as leader of term 4
I20260925 03:04:40.938946 24367 catalog_manager.cc:5495] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "eeab93abfdb141a895002ec9f312296c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } }
I20260925 03:04:40.952495 26305 raft_consensus.cc:729] T 4e5332a5d329493991dff994e70ea92c P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:04:40.952831 26246 raft_consensus.cc:3037] T a7aa47c48404465b94a36fd819fe3c7a P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:40.952967 24586 leader_election.cc:302] T c85f5d1d3f004cf5817bb594fd8e6469 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:04:40.953554 26270 consensus_queue.cc:260] T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:40.954046 26264 raft_consensus.cc:2781] T c85f5d1d3f004cf5817bb594fd8e6469 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader election won for term 4
I20260925 03:04:40.954674 26264 raft_consensus.cc:686] T c85f5d1d3f004cf5817bb594fd8e6469 P eeab93abfdb141a895002ec9f312296c [term 4 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:40.954053 24367 catalog_manager.cc:5495] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "eeab93abfdb141a895002ec9f312296c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } }
I20260925 03:04:40.954901 26305 consensus_queue.cc:260] T 4e5332a5d329493991dff994e70ea92c P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:40.955646 26264 consensus_queue.cc:237] T c85f5d1d3f004cf5817bb594fd8e6469 P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:40.958395 26305 raft_consensus.cc:3037] T 4e5332a5d329493991dff994e70ea92c P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:40.958570 26086 consensus_queue.cc:1046] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260925 03:04:40.957226 26209 consensus_queue.cc:1046] T 37a157996e944db2906110d87ecb0c54 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260925 03:04:40.960057 26209 raft_consensus.cc:3032] T 37a157996e944db2906110d87ecb0c54 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 LEADER]: Stepping down as leader of term 4
I20260925 03:04:40.960455 26209 raft_consensus.cc:729] T 37a157996e944db2906110d87ecb0c54 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:04:40.965097 24367 catalog_manager.cc:5495] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 reported cstate change: term changed from 3 to 4, leader changed from 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1) to eeab93abfdb141a895002ec9f312296c (127.22.193.3). New cstate: current_term: 4 leader_uuid: "eeab93abfdb141a895002ec9f312296c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: NON_VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: true } } }
I20260925 03:04:40.935981 26293 raft_consensus.cc:2781] T 74443a0a1dc34d88a2f42e3b722db405 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Leader election won for term 5
I20260925 03:04:40.969498 26293 raft_consensus.cc:686] T 74443a0a1dc34d88a2f42e3b722db405 P eeab93abfdb141a895002ec9f312296c [term 5 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:40.969694 26209 consensus_queue.cc:260] T 37a157996e944db2906110d87ecb0c54 P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:40.970516 26293 consensus_queue.cc:237] T 74443a0a1dc34d88a2f42e3b722db405 P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:40.960448 26270 raft_consensus.cc:3037] T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:40.973539 24466 raft_consensus.cc:1226] T 43366a0a2f9d4344bf4e37ffd290ef8a P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 2. Current term is 3. Ops: []
I20260925 03:04:40.959718 24436 leader_election.cc:302] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:40.975055 24540 raft_consensus.cc:2454] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 4.
I20260925 03:04:40.972136 24367 catalog_manager.cc:5495] T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5 reported cstate change: term changed from 1 to 2, leader changed from 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1) to eeab93abfdb141a895002ec9f312296c (127.22.193.3). New cstate: current_term: 2 leader_uuid: "eeab93abfdb141a895002ec9f312296c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } }
I20260925 03:04:40.976135 24547 raft_consensus.cc:1226] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Rejecting Update request from peer 1bc1f2e62ab44af995165bd5116a9d43 for earlier term 3. Current term is 4. Ops: []
I20260925 03:04:40.976933 24367 catalog_manager.cc:5495] T 550118bc1a0641ce9803bdbf43f2069f P 869b0a8575b54d08ae2817fad5dd42d5 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "869b0a8575b54d08ae2817fad5dd42d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: UNKNOWN } } }
I20260925 03:04:40.980926 26246 raft_consensus.cc:482] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:40.981724 24547 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33068 (request call id 102) took 1130 ms. Trace:
I20260925 03:04:40.981626 26246 raft_consensus.cc:504] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:40.982105 24547 rpcz_store.cc:276] 0925 03:04:39.851136 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:40.811095 (+959959us) service_pool.cc:227] Handling call
0925 03:04:40.975559 (+164464us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:40.980949 (+ 5390us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:40.981529 (+ 580us) spinlock_profiling.cc:243] Waited 140 ms on lock 0x7b54000fbed0. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764cfa09 00007f437b4112e9 00007f4374af7684 00007f4374af7612 00007f4374af7541 00007f4374af74bc 00007f4374af6772 00007f43748fa94c 00007f43748f9d86 00007f43748f972f
0925 03:04:40.981667 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":321261440}
I20260925 03:04:40.983588 26246 leader_election.cc:288] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:40.980311 24367 catalog_manager.cc:5495] T 0be4e0aa984f4f2b81a58e382e434ad5 P 869b0a8575b54d08ae2817fad5dd42d5 reported cstate change: term changed from 1 to 3, leader changed from 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1) to eeab93abfdb141a895002ec9f312296c (127.22.193.3). New cstate: current_term: 3 leader_uuid: "eeab93abfdb141a895002ec9f312296c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } }
I20260925 03:04:40.987318 24588 leader_election.cc:302] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:40.988368 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b5df303cd6a4d59aca5c18bbbff24ed" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:40.988677 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7993f9c3e2a44f408bbf5d1e395e2306" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:40.989461 24473 raft_consensus.cc:2379] T 7993f9c3e2a44f408bbf5d1e395e2306 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 4: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:04:40.989524 26348 raft_consensus.cc:2781] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Leader election won for term 5
I20260925 03:04:40.989858 26262 consensus_queue.cc:1035] T c88cbd39e01f44628a094af516be2833 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260925 03:04:40.990145 26348 raft_consensus.cc:686] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c [term 5 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:40.990624 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "420163a6cdce412ebc21c25fe3835638" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:40.991364 24473 raft_consensus.cc:2354] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 4. Current term is 5.
I20260925 03:04:40.991174 26348 consensus_queue.cc:237] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:40.991307 26262 consensus_queue.cc:1035] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
I20260925 03:04:40.994064 26083 consensus_queue.cc:1046] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260925 03:04:40.996706 24367 catalog_manager.cc:5495] T cc2df40db08649c9a75e90000b878559 P 869b0a8575b54d08ae2817fad5dd42d5 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "869b0a8575b54d08ae2817fad5dd42d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: UNKNOWN } } }
W20260925 03:04:41.000458 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 47576.8us
I20260925 03:04:40.984429 25942 raft_consensus.cc:504] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:40.995394 26262 raft_consensus.cc:3032] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [term 3 LEADER]: Stepping down as leader of term 3
I20260925 03:04:41.003341 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5332a5d329493991dff994e70ea92c" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:40.995390 26285 consensus_queue.cc:1035] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
I20260925 03:04:41.013165 24473 raft_consensus.cc:2396] T 4e5332a5d329493991dff994e70ea92c P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260925 03:04:41.014946 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74443a0a1dc34d88a2f42e3b722db405" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:41.017974 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b030d84c670b48d5bff337ccc0c77626" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:41.000983 26305 raft_consensus.cc:2726] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260925 03:04:41.003508 26209 raft_consensus.cc:3037] T 37a157996e944db2906110d87ecb0c54 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Advancing to term 5
W20260925 03:04:41.021612 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 72527.8us
I20260925 03:04:41.028223 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25102ba08e4e49388f6176badcd10a25" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:41.012938 26262 raft_consensus.cc:729] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:41.042466 24586 leader_election.cc:302] T 550118bc1a0641ce9803bdbf43f2069f P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:04:41.044884 26262 consensus_queue.cc:260] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:41.045855 24514 leader_election.cc:302] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 1bc1f2e62ab44af995165bd5116a9d43
I20260925 03:04:41.046077 24586 leader_election.cc:302] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:04:41.046499 26262 raft_consensus.cc:3037] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:41.047289 24586 leader_election.cc:302] T cc2df40db08649c9a75e90000b878559 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:04:41.047874 24514 leader_election.cc:302] T b4ebd2b043024a548005c8b6b00b0449 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 1bc1f2e62ab44af995165bd5116a9d43
I20260925 03:04:41.048882 26248 raft_consensus.cc:2781] T 550118bc1a0641ce9803bdbf43f2069f P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260925 03:04:41.049341 24473 raft_consensus.cc:1261] T 100c113a9fd84cc8adf2ec7b33ea0093 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260925 03:04:41.050920 24466 raft_consensus.cc:1261] T f22ddd7d1c444dc58f45a1fb0b3e02bc P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260925 03:04:41.036022 24367 catalog_manager.cc:5495] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 reported cstate change: term changed from 1 to 4. New cstate: current_term: 4 leader_uuid: "869b0a8575b54d08ae2817fad5dd42d5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260925 03:04:41.049259 26248 raft_consensus.cc:482] T 550118bc1a0641ce9803bdbf43f2069f P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting leader election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:41.060443 26241 raft_consensus.cc:3032] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 LEADER]: Stepping down as leader of term 3
I20260925 03:04:41.049536 26086 raft_consensus.cc:2741] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260925 03:04:41.049723 26348 raft_consensus.cc:2781] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Leader election won for term 5
I20260925 03:04:41.052057 26247 raft_consensus.cc:2781] T cc2df40db08649c9a75e90000b878559 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader election won for term 4
W20260925 03:04:41.091387 24436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 69557.9us
I20260925 03:04:41.093139 26247 raft_consensus.cc:686] T cc2df40db08649c9a75e90000b878559 P eeab93abfdb141a895002ec9f312296c [term 4 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:41.093266 24615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7993f9c3e2a44f408bbf5d1e395e2306" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:41.097947 24615 raft_consensus.cc:2396] T 7993f9c3e2a44f408bbf5d1e395e2306 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260925 03:04:41.097751 26247 consensus_queue.cc:237] T cc2df40db08649c9a75e90000b878559 P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:41.092427 24367 catalog_manager.cc:5495] T 1414dcfe53f64a78ae413531eb8ec117 P 869b0a8575b54d08ae2817fad5dd42d5 reported cstate change: term changed from 1 to 2, leader changed from 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1) to eeab93abfdb141a895002ec9f312296c (127.22.193.3). New cstate: current_term: 2 leader_uuid: "eeab93abfdb141a895002ec9f312296c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } }
I20260925 03:04:41.102283 24618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "420163a6cdce412ebc21c25fe3835638" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:41.081064 24465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5f9b503b0ab43999d582015a367441e" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:41.086565 26248 raft_consensus.cc:3037] T 550118bc1a0641ce9803bdbf43f2069f P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:41.103543 24618 raft_consensus.cc:2354] T 420163a6cdce412ebc21c25fe3835638 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 4. Current term is 5.
I20260925 03:04:41.086110 26241 raft_consensus.cc:729] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:04:41.104117 24436 leader_election.cc:302] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:41.110169 24514 leader_election.cc:302] T b030d84c670b48d5bff337ccc0c77626 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:41.116562 26357 raft_consensus.cc:2781] T b4ebd2b043024a548005c8b6b00b0449 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260925 03:04:41.129506 26357 raft_consensus.cc:482] T b4ebd2b043024a548005c8b6b00b0449 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1bc1f2e62ab44af995165bd5116a9d43)
I20260925 03:04:41.129652 24436 leader_election.cc:302] T 9e8af2641e124f9589c5ccfdac785f63 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:41.132622 24436 leader_election.cc:302] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: 1bc1f2e62ab44af995165bd5116a9d43, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:41.133956 24615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25102ba08e4e49388f6176badcd10a25" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:41.109161 24465 raft_consensus.cc:2454] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 5.
W20260925 03:04:41.141692 24511 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 82611.2us
I20260925 03:04:41.144706 24464 raft_consensus.cc:1261] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260925 03:04:41.137341 26357 raft_consensus.cc:3037] T b4ebd2b043024a548005c8b6b00b0449 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:41.106307 26325 consensus_queue.cc:1046] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.030s
I20260925 03:04:41.143517 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b8e0d260a884ecdb20f79de46c36b19" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:41.109401 26086 consensus_queue.cc:1046] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.029s
I20260925 03:04:41.147140 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "420163a6cdce412ebc21c25fe3835638" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:41.150411 24540 raft_consensus.cc:3037] T 420163a6cdce412ebc21c25fe3835638 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:41.160771 24548 raft_consensus.cc:1261] T 442d4a29b1d743599f4b6b4871e26668 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260925 03:04:41.165397 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7aa47c48404465b94a36fd819fe3c7a" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:41.165030 26241 consensus_queue.cc:260] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:41.167980 26241 raft_consensus.cc:3037] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Advancing to term 4
W20260925 03:04:41.169243 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 67803us
I20260925 03:04:41.170181 24514 leader_election.cc:302] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters: 1bc1f2e62ab44af995165bd5116a9d43
I20260925 03:04:41.173918 24511 leader_election.cc:398] T 420163a6cdce412ebc21c25fe3835638 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 election: Vote denied by peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105) with higher term. Message: Invalid argument: T 420163a6cdce412ebc21c25fe3835638 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 4. Current term is 5.
I20260925 03:04:41.176712 24618 raft_consensus.cc:1261] T 43366a0a2f9d4344bf4e37ffd290ef8a P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260925 03:04:41.176039 24547 raft_consensus.cc:3037] T a7aa47c48404465b94a36fd819fe3c7a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:41.174875 24511 leader_election.cc:401] T 420163a6cdce412ebc21c25fe3835638 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
W20260925 03:04:41.176292 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 71202.4us
I20260925 03:04:41.174374 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f68804eb5a244d87bb395d50683c6012" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:41.175729 25942 leader_election.cc:288] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:41.161782 26083 raft_consensus.cc:2932] T 367277daf8d04778b3fa4ed94d71580d P eeab93abfdb141a895002ec9f312296c [term 4 LEADER]: Committing config change with OpId 4.5: config changed from index 2 to 5, 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:41.199777 24545 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 1297) took 1008 ms. Trace:
I20260925 03:04:41.200261 24545 rpcz_store.cc:276] 0925 03:04:40.191558 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:40.833755 (+642197us) service_pool.cc:227] Handling call
0925 03:04:40.853943 (+ 20188us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:04:40.906757 (+ 52814us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:40.906785 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:04:40.907272 (+ 487us) log.cc:842] Serialized 26 byte log entry
0925 03:04:40.974045 (+ 66773us) raft_consensus.cc:1631] Marking committed up to 5
0925 03:04:41.069160 (+ 95115us) log.cc:842] Serialized 14 byte log entry
0925 03:04:41.142003 (+ 72843us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:41.142072 (+ 69us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:04:41.199499 (+ 57427us) raft_consensus.cc:1671] finished
0925 03:04:41.199523 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:41.199672 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":199536,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":38,"raft.queue_time_us":111644,"raft.run_cpu_time_us":206,"raft.run_wall_time_us":208,"thread_start_us":204653,"threads_started":3,"wal-append.queue_time_us":186287}
I20260925 03:04:41.230719 25194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e92f377a79354a6ebb4eec67303c13e3" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43"
I20260925 03:04:41.201108 24541 raft_consensus.cc:2454] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 3.
I20260925 03:04:41.239326 24540 raft_consensus.cc:2454] T 420163a6cdce412ebc21c25fe3835638 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 5.
W20260925 03:04:41.248085 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 115460us
I20260925 03:04:41.250841 24547 raft_consensus.cc:2454] T a7aa47c48404465b94a36fd819fe3c7a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 3.
I20260925 03:04:41.250037 26305 raft_consensus.cc:2726] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260925 03:04:41.262193 24588 leader_election.cc:302] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
W20260925 03:04:41.272853 24436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 140297us
I20260925 03:04:41.290849 26362 raft_consensus.cc:2726] T b030d84c670b48d5bff337ccc0c77626 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260925 03:04:41.291602 26293 consensus_queue.cc:1035] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20260925 03:04:41.292881 26364 log.cc:924] Time spent T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5: Append to log took a long time: real 0.059s user 0.001s sys 0.000s
I20260925 03:04:41.291828 26363 raft_consensus.cc:482] T a4abe0ea757e4886be4945c5001e529a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:41.296295 26365 raft_consensus.cc:2726] T 9e8af2641e124f9589c5ccfdac785f63 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260925 03:04:41.302115 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 132093us
I20260925 03:04:41.281304 26086 raft_consensus.cc:3032] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 LEADER]: Stepping down as leader of term 4
I20260925 03:04:41.275799 26348 raft_consensus.cc:686] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c [term 5 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:41.289026 26325 raft_consensus.cc:3032] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 LEADER]: Stepping down as leader of term 4
I20260925 03:04:41.291622 26362 raft_consensus.cc:2726] T 420163a6cdce412ebc21c25fe3835638 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105) with higher term. Message: Invalid argument: T 420163a6cdce412ebc21c25fe3835638 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 4. Current term is 5.
I20260925 03:04:41.311717 24547 raft_consensus.cc:1226] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 3. Current term is 4. Ops: []
I20260925 03:04:41.313768 24541 raft_consensus.cc:1261] T 8e5819c15eac4544a57e5ee6c9d4f117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260925 03:04:41.306491 26086 raft_consensus.cc:729] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:04:41.317855 26378 consensus_queue.cc:1035] T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.007s
I20260925 03:04:41.319106 26368 raft_consensus.cc:2741] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260925 03:04:41.307804 26325 raft_consensus.cc:729] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:04:41.302929 26363 raft_consensus.cc:504] T a4abe0ea757e4886be4945c5001e529a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:41.335815 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f1fac028a144058b92331c0d9b44ebf" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:41.305904 26264 consensus_queue.cc:1035] T 100c113a9fd84cc8adf2ec7b33ea0093 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.011s
I20260925 03:04:41.355417 26209 raft_consensus.cc:2726] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260925 03:04:41.366206 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 83623.2us
I20260925 03:04:41.370189 24547 raft_consensus.cc:1226] T a7aa47c48404465b94a36fd819fe3c7a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260925 03:04:41.377547 26380 raft_consensus.cc:2781] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260925 03:04:41.378087 26380 raft_consensus.cc:482] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:41.378523 26380 raft_consensus.cc:3037] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Advancing to term 6
W20260925 03:04:41.378001 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 67341.1us
I20260925 03:04:41.382237 24514 leader_election.cc:302] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:41.382709 26363 leader_election.cc:288] T a4abe0ea757e4886be4945c5001e529a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:41.383177 26375 raft_consensus.cc:3037] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:41.386044 25194 raft_consensus.cc:1261] T 8ac466793d094325ace224708b2158ed P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 6. (term mismatch)
I20260925 03:04:41.386404 25194 pending_rounds.cc:76] T 8ac466793d094325ace224708b2158ed P 1bc1f2e62ab44af995165bd5116a9d43: Aborting all ops after (but not including) 5
I20260925 03:04:41.386612 25194 pending_rounds.cc:97] T 8ac466793d094325ace224708b2158ed P 1bc1f2e62ab44af995165bd5116a9d43: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20260925 03:04:41.386889 25194 raft_consensus.cc:2866] T 8ac466793d094325ace224708b2158ed P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260925 03:04:41.386747 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5278a60f97da4e10aecd50a8d9a9a43c" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:41.389004 24541 raft_consensus.cc:2379] T 5278a60f97da4e10aecd50a8d9a9a43c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c in current term 3: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:04:41.391598 26368 consensus_queue.cc:1035] T 43366a0a2f9d4344bf4e37ffd290ef8a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260925 03:04:41.395882 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "420163a6cdce412ebc21c25fe3835638" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:41.396762 24543 raft_consensus.cc:2379] T 420163a6cdce412ebc21c25fe3835638 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c in current term 5: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:04:41.397326 24548 raft_consensus.cc:1261] T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260925 03:04:41.398819 24543 raft_consensus.cc:1261] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260925 03:04:41.400223 24436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 82822.2us
I20260925 03:04:41.403857 24542 raft_consensus.cc:1261] T 37a157996e944db2906110d87ecb0c54 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260925 03:04:41.405936 24465 raft_consensus.cc:3032] T ac9357d144aa49789faaf31fe5e6bf89 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 LEADER]: Stepping down as leader of term 1
I20260925 03:04:41.406312 24465 raft_consensus.cc:729] T ac9357d144aa49789faaf31fe5e6bf89 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:04:41.403919 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b030d84c670b48d5bff337ccc0c77626" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:41.353860 26373 raft_consensus.cc:482] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:41.407601 24546 raft_consensus.cc:3037] T b030d84c670b48d5bff337ccc0c77626 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:04:41.404742 24539 raft_consensus.cc:1261] T c88cbd39e01f44628a094af516be2833 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260925 03:04:41.413085 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4abe0ea757e4886be4945c5001e529a" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:41.416738 24466 raft_consensus.cc:3032] T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 LEADER]: Stepping down as leader of term 4
I20260925 03:04:41.417111 24466 raft_consensus.cc:729] T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:04:41.417788 24466 consensus_queue.cc:260] T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:41.418946 24466 raft_consensus.cc:3037] T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:41.421141 24618 raft_consensus.cc:1226] T 21c43af5e6844fdaacc8c625fa01fd8d P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 2. Current term is 3. Ops: []
I20260925 03:04:41.422581 26377 consensus_queue.cc:1046] T edcf6e6a422d4d198d4fbb0f7d3d845d P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260925 03:04:41.423830 26377 raft_consensus.cc:3032] T edcf6e6a422d4d198d4fbb0f7d3d845d P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 LEADER]: Stepping down as leader of term 1
I20260925 03:04:41.420969 24464 raft_consensus.cc:1261] T 7b8e0d260a884ecdb20f79de46c36b19 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260925 03:04:41.424193 26377 raft_consensus.cc:729] T edcf6e6a422d4d198d4fbb0f7d3d845d P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:04:41.424918 25194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4abe0ea757e4886be4945c5001e529a" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:41.426126 24547 raft_consensus.cc:1226] T 25102ba08e4e49388f6176badcd10a25 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 1. Current term is 2. Ops: []
I20260925 03:04:41.428323 24546 raft_consensus.cc:2454] T b030d84c670b48d5bff337ccc0c77626 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 6.
I20260925 03:04:41.429337 24543 raft_consensus.cc:1261] T f22ddd7d1c444dc58f45a1fb0b3e02bc P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260925 03:04:41.431564 24437 leader_election.cc:302] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:41.440990 24464 raft_consensus.cc:1226] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 3. Current term is 4. Ops: []
I20260925 03:04:41.445647 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5f9b503b0ab43999d582015a367441e" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
W20260925 03:04:41.446681 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 151174us
W20260925 03:04:41.450075 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 67967.9us
I20260925 03:04:41.399049 26241 consensus_queue.cc:1035] T 8ac466793d094325ace224708b2158ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.045s
I20260925 03:04:41.461418 24437 leader_election.cc:302] T 1414dcfe53f64a78ae413531eb8ec117 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:41.442355 26369 raft_consensus.cc:2726] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260925 03:04:41.473343 24465 consensus_queue.cc:260] T ac9357d144aa49789faaf31fe5e6bf89 P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:41.487510 26376 raft_consensus.cc:482] T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:41.460227 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:41.488190 26376 raft_consensus.cc:504] T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:41.490484 26376 leader_election.cc:288] T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:41.491787 24542 raft_consensus.cc:1261] T 100c113a9fd84cc8adf2ec7b33ea0093 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260925 03:04:41.487914 24437 leader_election.cc:302] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:41.502439 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e92f377a79354a6ebb4eec67303c13e3" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:41.503266 24546 raft_consensus.cc:3037] T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:41.505204 24465 raft_consensus.cc:3037] T ac9357d144aa49789faaf31fe5e6bf89 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:41.493147 26379 raft_consensus.cc:2726] T 1414dcfe53f64a78ae413531eb8ec117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260925 03:04:41.483491 24464 raft_consensus.cc:2932] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Committing config change with OpId 4.5: config changed from index 2 to 5, 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:41.502111 25190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e068a0f80941dc9138cee063f53bab" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:41.509174 24436 leader_election.cc:302] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters: 1bc1f2e62ab44af995165bd5116a9d43
I20260925 03:04:41.509402 24514 leader_election.cc:302] T 0be4e0aa984f4f2b81a58e382e434ad5 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:41.522042 24514 leader_election.cc:302] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:04:41.523710 26380 raft_consensus.cc:504] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:41.524952 26368 raft_consensus.cc:2726] T 0be4e0aa984f4f2b81a58e382e434ad5 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260925 03:04:41.501039 26377 consensus_queue.cc:260] T edcf6e6a422d4d198d4fbb0f7d3d845d P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:41.512785 26370 consensus_queue.cc:1046] T c85f5d1d3f004cf5817bb594fd8e6469 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260925 03:04:41.512982 25979 consensus_queue.cc:1035] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260925 03:04:41.525799 26380 leader_election.cc:288] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 6 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:41.526314 26377 raft_consensus.cc:3037] T edcf6e6a422d4d198d4fbb0f7d3d845d P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:41.517297 24437 leader_election.cc:398] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 election: Vote denied by peer a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389) with higher term. Message: Invalid argument: T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 4. Current term is 5.
I20260925 03:04:41.518678 24541 raft_consensus.cc:2379] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c in current term 5: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:04:41.528544 26375 raft_consensus.cc:2726] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260925 03:04:41.530385 24544 raft_consensus.cc:2932] T 367277daf8d04778b3fa4ed94d71580d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Committing config change with OpId 4.5: config changed from index 2 to 5, 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:41.527745 24437 leader_election.cc:302] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:04:41.532404 24620 raft_consensus.cc:1226] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Rejecting Update request from peer 1bc1f2e62ab44af995165bd5116a9d43 for earlier term 3. Current term is 4. Ops: []
I20260925 03:04:41.532604 24541 raft_consensus.cc:1261] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260925 03:04:41.524564 24514 leader_election.cc:302] T f22ddd7d1c444dc58f45a1fb0b3e02bc P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:41.534125 24615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e068a0f80941dc9138cee063f53bab" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:41.534507 26386 consensus_queue.cc:1046] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260925 03:04:41.454818 26325 consensus_queue.cc:260] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:41.461704 26373 raft_consensus.cc:504] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:41.538388 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5f9b503b0ab43999d582015a367441e" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:41.539063 24472 raft_consensus.cc:3037] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:04:41.539463 24514 leader_election.cc:302] T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:04:41.539736 26373 leader_election.cc:288] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:41.540032 24511 leader_election.cc:302] T a4abe0ea757e4886be4945c5001e529a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 1bc1f2e62ab44af995165bd5116a9d43, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:41.540253 26373 raft_consensus.cc:2726] T f22ddd7d1c444dc58f45a1fb0b3e02bc P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260925 03:04:41.541054 26373 consensus_queue.cc:1046] T 21c43af5e6844fdaacc8c625fa01fd8d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260925 03:04:41.541848 26373 raft_consensus.cc:2726] T a4abe0ea757e4886be4945c5001e529a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260925 03:04:41.542505 26363 raft_consensus.cc:2781] T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260925 03:04:41.542959 26363 raft_consensus.cc:482] T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:41.543345 26363 raft_consensus.cc:3037] T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:41.543663 25196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71f461a6e2ff434993b78a2d077f0abd" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:41.544297 25196 raft_consensus.cc:2454] T 71f461a6e2ff434993b78a2d077f0abd P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 4.
I20260925 03:04:41.522264 24540 raft_consensus.cc:1261] T ac9357d144aa49789faaf31fe5e6bf89 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260925 03:04:41.505220 26365 raft_consensus.cc:2726] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260925 03:04:41.538290 26337 consensus_queue.cc:1046] T e92f377a79354a6ebb4eec67303c13e3 P 1bc1f2e62ab44af995165bd5116a9d43 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260925 03:04:41.546344 24472 raft_consensus.cc:2454] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 6.
I20260925 03:04:41.550130 24511 leader_election.cc:302] T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 1bc1f2e62ab44af995165bd5116a9d43, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:41.552307 24588 leader_election.cc:302] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:41.556514 24586 leader_election.cc:302] T 5278a60f97da4e10aecd50a8d9a9a43c P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:41.562258 24511 leader_election.cc:302] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:04:41.567607 24546 raft_consensus.cc:2454] T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 4.
I20260925 03:04:41.414158 26086 consensus_queue.cc:260] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:41.542068 26381 raft_consensus.cc:3032] T 21c43af5e6844fdaacc8c625fa01fd8d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 LEADER]: Stepping down as leader of term 2
W20260925 03:04:41.576316 24436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 69722.3us
I20260925 03:04:41.577411 26362 raft_consensus.cc:2726] T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260925 03:04:41.580255 26385 log.cc:924] Time spent T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5: Append to log took a long time: real 0.085s user 0.000s sys 0.000s
I20260925 03:04:41.560030 26293 consensus_queue.cc:1035] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.025s
I20260925 03:04:41.573837 26375 raft_consensus.cc:2781] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260925 03:04:41.534435 25194 raft_consensus.cc:2932] T 367277daf8d04778b3fa4ed94d71580d P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Committing config change with OpId 4.5: config changed from index 2 to 5, 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:41.585035 26379 raft_consensus.cc:2741] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260925 03:04:41.585278 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5f9b503b0ab43999d582015a367441e" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:41.580297 24514 leader_election.cc:398] T 100c113a9fd84cc8adf2ec7b33ea0093 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 election: Vote denied by peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187) with higher term. Message: Invalid argument: T 100c113a9fd84cc8adf2ec7b33ea0093 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 2. Current term is 3.
I20260925 03:04:41.617274 24542 raft_consensus.cc:3037] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:04:41.463276 26348 consensus_queue.cc:237] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:41.536029 26366 raft_consensus.cc:3032] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c [term 3 LEADER]: Stepping down as leader of term 3
I20260925 03:04:41.536041 26325 raft_consensus.cc:3037] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Advancing to term 5
W20260925 03:04:41.580806 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 53135.6us
I20260925 03:04:41.594977 24586 leader_election.cc:302] T 420163a6cdce412ebc21c25fe3835638 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:41.598409 24466 raft_consensus.cc:1261] T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260925 03:04:41.603215 24472 raft_consensus.cc:3032] T c85f5d1d3f004cf5817bb594fd8e6469 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 LEADER]: Stepping down as leader of term 3
I20260925 03:04:41.585163 26209 raft_consensus.cc:2781] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260925 03:04:41.617363 24514 leader_election.cc:401] T 100c113a9fd84cc8adf2ec7b33ea0093 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260925 03:04:41.509130 26357 raft_consensus.cc:504] T b4ebd2b043024a548005c8b6b00b0449 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:41.624395 24472 raft_consensus.cc:729] T c85f5d1d3f004cf5817bb594fd8e6469 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:04:41.587251 26381 raft_consensus.cc:729] T 21c43af5e6844fdaacc8c625fa01fd8d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:04:41.600010 26375 raft_consensus.cc:482] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:41.639261 24615 raft_consensus.cc:1261] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260925 03:04:41.635363 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71f461a6e2ff434993b78a2d077f0abd" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:41.619320 26264 raft_consensus.cc:2726] T 5278a60f97da4e10aecd50a8d9a9a43c P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260925 03:04:41.644598 24620 raft_consensus.cc:2454] T 71f461a6e2ff434993b78a2d077f0abd P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 4.
I20260925 03:04:41.620152 26366 raft_consensus.cc:729] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:41.623180 26209 raft_consensus.cc:482] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:41.659071 24437 leader_election.cc:398] T 7b8e0d260a884ecdb20f79de46c36b19 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 election: Vote denied by peer a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389) with higher term. Message: Invalid argument: T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 3. Current term is 4.
I20260925 03:04:41.659569 24437 leader_election.cc:401] T 7b8e0d260a884ecdb20f79de46c36b19 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260925 03:04:41.641361 26375 raft_consensus.cc:3037] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:41.640331 26285 raft_consensus.cc:2726] T 420163a6cdce412ebc21c25fe3835638 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260925 03:04:41.663813 24436 leader_election.cc:302] T 11bb46bce3564841b7674e6a693bd2e6 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:41.646770 26337 raft_consensus.cc:3032] T e92f377a79354a6ebb4eec67303c13e3 P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 LEADER]: Stepping down as leader of term 3
I20260925 03:04:41.644867 26241 raft_consensus.cc:2726] T 100c113a9fd84cc8adf2ec7b33ea0093 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187) with higher term. Message: Invalid argument: T 100c113a9fd84cc8adf2ec7b33ea0093 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 2. Current term is 3.
I20260925 03:04:41.631568 26370 consensus_queue.cc:1046] T cc2df40db08649c9a75e90000b878559 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.029s
I20260925 03:04:41.658293 26209 raft_consensus.cc:3037] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:41.635824 25979 consensus_queue.cc:1035] T 74443a0a1dc34d88a2f42e3b722db405 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.026s
W20260925 03:04:41.669169 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 89000.4us
I20260925 03:04:41.665302 26245 consensus_queue.cc:1035] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.011s
I20260925 03:04:41.662994 24472 consensus_queue.cc:260] T c85f5d1d3f004cf5817bb594fd8e6469 P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:41.675555 26372 raft_consensus.cc:2781] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Leader election won for term 6
I20260925 03:04:41.675951 26368 raft_consensus.cc:2781] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260925 03:04:41.678550 24473 raft_consensus.cc:1261] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260925 03:04:41.644363 26247 consensus_queue.cc:1035] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.022s
I20260925 03:04:41.670470 26270 consensus_queue.cc:1035] T 37a157996e944db2906110d87ecb0c54 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260925 03:04:41.666529 26337 raft_consensus.cc:729] T e92f377a79354a6ebb4eec67303c13e3 P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1bc1f2e62ab44af995165bd5116a9d43, State: Running, Role: LEADER
I20260925 03:04:41.671074 26365 raft_consensus.cc:2726] T 11bb46bce3564841b7674e6a693bd2e6 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260925 03:04:41.686395 24514 leader_election.cc:302] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:04:41.653649 26362 consensus_queue.cc:1035] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.027s
I20260925 03:04:41.681584 26372 raft_consensus.cc:686] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [term 6 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:41.690539 24437 leader_election.cc:302] T 06f4a7eaf6534e4986c9fd35911038e4 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:04:41.690531 26372 consensus_queue.cc:237] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:41.685105 26366 consensus_queue.cc:260] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:41.694073 26366 raft_consensus.cc:3037] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Advancing to term 4
W20260925 03:04:41.684787 26248 consensus_meta.cc:287] T 550118bc1a0641ce9803bdbf43f2069f P eeab93abfdb141a895002ec9f312296c: Time spent flushing consensus metadata: real 0.557s user 0.011s sys 0.001s
I20260925 03:04:41.725414 24542 raft_consensus.cc:2454] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 6.
I20260925 03:04:41.728480 24436 leader_election.cc:302] T 21c43af5e6844fdaacc8c625fa01fd8d P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:41.728526 26370 raft_consensus.cc:2781] T 06f4a7eaf6534e4986c9fd35911038e4 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260925 03:04:41.730572 26370 raft_consensus.cc:482] T 06f4a7eaf6534e4986c9fd35911038e4 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:41.730932 26370 raft_consensus.cc:3037] T 06f4a7eaf6534e4986c9fd35911038e4 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:41.735517 26305 raft_consensus.cc:2781] T 21c43af5e6844fdaacc8c625fa01fd8d P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 03:04:41.736588 24618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be4e0aa984f4f2b81a58e382e434ad5" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:41.737552 25193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5aec38e66a947c681e6c2a72b05e6ca" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43"
I20260925 03:04:41.738395 25193 raft_consensus.cc:2379] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 in current term 6: Already voted for candidate 1bc1f2e62ab44af995165bd5116a9d43 in this term.
I20260925 03:04:41.741987 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35992316cb8d4c028333388eff02ecde" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:41.742810 24622 raft_consensus.cc:2379] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 in current term 5: Already voted for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in this term.
I20260925 03:04:41.747529 24514 leader_election.cc:302] T 37a157996e944db2906110d87ecb0c54 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:41.736529 26305 raft_consensus.cc:686] T 21c43af5e6844fdaacc8c625fa01fd8d P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 LEADER]: Becoming Leader. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:04:41.750801 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:41.722553 26373 raft_consensus.cc:2781] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260925 03:04:41.757160 24465 raft_consensus.cc:1261] T ac9357d144aa49789faaf31fe5e6bf89 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260925 03:04:41.758271 24619 raft_consensus.cc:2379] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 in current term 5: Already voted for candidate eeab93abfdb141a895002ec9f312296c in this term.
I20260925 03:04:41.688099 26363 raft_consensus.cc:504] T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:04:41.759163 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 96080.4us
I20260925 03:04:41.759665 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5aec38e66a947c681e6c2a72b05e6ca" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:41.763092 24620 raft_consensus.cc:2396] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260925 03:04:41.763659 24614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25102ba08e4e49388f6176badcd10a25" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:41.764083 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c81a1b3b616492ab6c964f45af36077" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:41.766355 24622 raft_consensus.cc:1226] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 5. Current term is 6. Ops: []
I20260925 03:04:41.661552 26381 consensus_queue.cc:260] T 21c43af5e6844fdaacc8c625fa01fd8d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:41.727474 26248 raft_consensus.cc:504] T 550118bc1a0641ce9803bdbf43f2069f P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:41.775849 24615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b627ecb4a5384ef392aa695e06599c39" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:41.756345 26373 raft_consensus.cc:482] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:41.755990 25196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6b6ec010caf4689b8c90449764f5cf9" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43"
I20260925 03:04:41.779124 25165 leader_election.cc:302] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
W20260925 03:04:41.781386 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 98345.7us
I20260925 03:04:41.730224 26383 consensus_queue.cc:1035] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.057s
I20260925 03:04:41.782858 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7aa47c48404465b94a36fd819fe3c7a" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:41.784832 24622 raft_consensus.cc:2396] T a7aa47c48404465b94a36fd819fe3c7a P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260925 03:04:41.783310 24618 raft_consensus.cc:2454] T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 4.
I20260925 03:04:41.785171 24472 raft_consensus.cc:3037] T c85f5d1d3f004cf5817bb594fd8e6469 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:41.769711 26379 raft_consensus.cc:2726] T 7b8e0d260a884ecdb20f79de46c36b19 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389) with higher term. Message: Invalid argument: T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 3. Current term is 4.
I20260925 03:04:41.788479 26381 raft_consensus.cc:3037] T 21c43af5e6844fdaacc8c625fa01fd8d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:41.787266 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "420163a6cdce412ebc21c25fe3835638" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:41.789615 24619 raft_consensus.cc:2379] T 420163a6cdce412ebc21c25fe3835638 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 in current term 5: Already voted for candidate eeab93abfdb141a895002ec9f312296c in this term.
I20260925 03:04:41.791209 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c81a1b3b616492ab6c964f45af36077" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:41.791855 25193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4ebd2b043024a548005c8b6b00b0449" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43"
I20260925 03:04:41.792770 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367277daf8d04778b3fa4ed94d71580d" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:41.794094 26374 raft_consensus.cc:3032] T cc2df40db08649c9a75e90000b878559 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 LEADER]: Stepping down as leader of term 3
I20260925 03:04:41.794515 26374 raft_consensus.cc:729] T cc2df40db08649c9a75e90000b878559 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:04:41.794770 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "293ef69caf514ac98510a68495f54eda" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:41.795329 24470 raft_consensus.cc:3037] T 293ef69caf514ac98510a68495f54eda P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:41.769975 26401 raft_consensus.cc:482] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:41.799182 26401 raft_consensus.cc:504] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:41.800312 26368 raft_consensus.cc:482] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:41.791939 24468 raft_consensus.cc:2379] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1bc1f2e62ab44af995165bd5116a9d43 in current term 4: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:04:41.801373 26401 leader_election.cc:288] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
W20260925 03:04:41.805379 26390 log.cc:924] Time spent T c88cbd39e01f44628a094af516be2833 P 869b0a8575b54d08ae2817fad5dd42d5: Append to log took a long time: real 0.066s user 0.001s sys 0.000s
I20260925 03:04:41.805738 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43366a0a2f9d4344bf4e37ffd290ef8a" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
W20260925 03:04:41.822824 25165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 69039.6us
I20260925 03:04:41.814900 24470 raft_consensus.cc:2454] T 293ef69caf514ac98510a68495f54eda P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 4.
I20260925 03:04:41.815639 24618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f68804eb5a244d87bb395d50683c6012" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:41.817556 24465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1e85248d3b04b979dd080caeeb02fdb" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:41.782577 26373 raft_consensus.cc:3037] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:41.782301 26357 leader_election.cc:288] T b4ebd2b043024a548005c8b6b00b0449 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:41.829630 24465 raft_consensus.cc:3037] T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:41.755782 26337 consensus_queue.cc:260] T e92f377a79354a6ebb4eec67303c13e3 P 1bc1f2e62ab44af995165bd5116a9d43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:41.833148 26337 raft_consensus.cc:3037] T e92f377a79354a6ebb4eec67303c13e3 P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:41.837231 26375 raft_consensus.cc:504] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:41.839308 26086 raft_consensus.cc:3037] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:41.817301 26368 raft_consensus.cc:3037] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:41.816576 26305 consensus_queue.cc:237] T 21c43af5e6844fdaacc8c625fa01fd8d P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:41.846016 24437 leader_election.cc:302] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:04:41.853096 24618 raft_consensus.cc:1226] T 550118bc1a0641ce9803bdbf43f2069f P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 3. Current term is 4. Ops: []
I20260925 03:04:41.853087 24621 raft_consensus.cc:2454] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 4.
W20260925 03:04:41.865202 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 139117us
I20260925 03:04:41.831617 26362 raft_consensus.cc:2726] T 37a157996e944db2906110d87ecb0c54 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260925 03:04:41.872049 24514 leader_election.cc:302] T 3d89396576884f7c8901dba54f783947 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:41.875031 26400 raft_consensus.cc:2726] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260925 03:04:41.875284 26401 consensus_queue.cc:1046] T e6b6ec010caf4689b8c90449764f5cf9 P 1bc1f2e62ab44af995165bd5116a9d43 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260925 03:04:41.875802 26374 consensus_queue.cc:260] T cc2df40db08649c9a75e90000b878559 P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:41.876464 26400 raft_consensus.cc:3032] T e6b6ec010caf4689b8c90449764f5cf9 P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 LEADER]: Stepping down as leader of term 3
I20260925 03:04:41.876950 26400 raft_consensus.cc:729] T e6b6ec010caf4689b8c90449764f5cf9 P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1bc1f2e62ab44af995165bd5116a9d43, State: Running, Role: LEADER
I20260925 03:04:41.877817 26400 consensus_queue.cc:260] T e6b6ec010caf4689b8c90449764f5cf9 P 1bc1f2e62ab44af995165bd5116a9d43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:41.863354 24615 raft_consensus.cc:2454] T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 5.
W20260925 03:04:41.885193 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 45675.2us
I20260925 03:04:41.863353 26367 consensus_queue.cc:1035] T ac9357d144aa49789faaf31fe5e6bf89 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.032s
I20260925 03:04:41.888496 24437 leader_election.cc:302] T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:04:41.901595 24472 raft_consensus.cc:1261] T c85f5d1d3f004cf5817bb594fd8e6469 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260925 03:04:41.903000 26371 raft_consensus.cc:2781] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader election won for term 4
W20260925 03:04:41.903234 26369 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 4 current term: 4 is higher.
I20260925 03:04:41.904538 26285 raft_consensus.cc:482] T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:41.904942 24437 leader_election.cc:302] T 3d89396576884f7c8901dba54f783947 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:41.905193 26375 leader_election.cc:288] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 election: Requested vote from peers eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:41.905071 26285 raft_consensus.cc:504] T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
W20260925 03:04:41.909642 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 43619.8us
I20260925 03:04:41.910728 24465 raft_consensus.cc:2454] T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 4.
I20260925 03:04:41.914080 24615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71f461a6e2ff434993b78a2d077f0abd" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:41.914587 24615 raft_consensus.cc:3037] T 71f461a6e2ff434993b78a2d077f0abd P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:41.887610 26357 raft_consensus.cc:2726] T 3d89396576884f7c8901dba54f783947 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260925 03:04:41.916157 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 150391us
I20260925 03:04:41.871028 26270 consensus_queue.cc:1035] T c88cbd39e01f44628a094af516be2833 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.035s
W20260925 03:04:41.917277 24436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.4:43979, user_credentials={real_user=slave}} blocked reactor thread for 123119us
I20260925 03:04:41.919034 24586 leader_election.cc:302] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:04:41.904443 26248 leader_election.cc:288] T 550118bc1a0641ce9803bdbf43f2069f P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:41.877172 26374 raft_consensus.cc:3037] T cc2df40db08649c9a75e90000b878559 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:41.929433 26409 raft_consensus.cc:2781] T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260925 03:04:41.933000 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "550118bc1a0641ce9803bdbf43f2069f" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:41.920531 24514 leader_election.cc:302] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:04:41.863394 26209 raft_consensus.cc:504] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:41.950109 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e8af2641e124f9589c5ccfdac785f63" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
W20260925 03:04:41.952225 25165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 82261.8us
W20260925 03:04:41.969100 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 53160.4us
I20260925 03:04:41.969604 24471 raft_consensus.cc:1226] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 4. Current term is 5. Ops: []
I20260925 03:04:41.970311 24615 raft_consensus.cc:2454] T 71f461a6e2ff434993b78a2d077f0abd P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 5.
I20260925 03:04:41.940325 25194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71f461a6e2ff434993b78a2d077f0abd" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43"
I20260925 03:04:41.940950 26363 leader_election.cc:288] T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:41.915182 26383 consensus_queue.cc:1035] T c85f5d1d3f004cf5817bb594fd8e6469 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.036s
I20260925 03:04:41.971576 24511 leader_election.cc:302] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:41.941793 26409 raft_consensus.cc:482] T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:41.972770 24436 leader_election.cc:302] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: 1bc1f2e62ab44af995165bd5116a9d43, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:41.973839 24514 leader_election.cc:302] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:41.945179 26086 raft_consensus.cc:2726] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260925 03:04:41.946090 26248 raft_consensus.cc:2781] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Leader election won for term 6
I20260925 03:04:41.981180 24586 leader_election.cc:302] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
W20260925 03:04:41.984869 26410 log.cc:924] Time spent T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.101s user 0.001s sys 0.000s
I20260925 03:04:41.934659 26368 raft_consensus.cc:504] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:41.979138 25194 raft_consensus.cc:3037] T 71f461a6e2ff434993b78a2d077f0abd P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:41.978579 26357 raft_consensus.cc:2781] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader election won for term 5
I20260925 03:04:42.012913 25165 leader_election.cc:302] T 367277daf8d04778b3fa4ed94d71580d P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:42.013273 24514 leader_election.cc:302] T ac9357d144aa49789faaf31fe5e6bf89 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:41.978199 26409 raft_consensus.cc:3037] T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:42.017269 25165 leader_election.cc:302] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:42.019506 24615 raft_consensus.cc:1261] T 442d4a29b1d743599f4b6b4871e26668 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260925 03:04:41.972673 25942 consensus_queue.cc:1046] T 25102ba08e4e49388f6176badcd10a25 P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.040s
W20260925 03:04:42.029371 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 48289.2us
I20260925 03:04:42.032004 24514 leader_election.cc:302] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:04:42.008200 26363 raft_consensus.cc:2726] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260925 03:04:42.032302 25165 leader_election.cc:302] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5; no voters:
W20260925 03:04:42.040745 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 55731.5us
I20260925 03:04:42.041973 26417 raft_consensus.cc:2726] T 367277daf8d04778b3fa4ed94d71580d P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260925 03:04:42.042994 26404 log.cc:924] Time spent T 8ac466793d094325ace224708b2158ed P 1bc1f2e62ab44af995165bd5116a9d43: Append to log took a long time: real 0.076s user 0.001s sys 0.000s
I20260925 03:04:42.045406 26381 raft_consensus.cc:2726] T ac9357d144aa49789faaf31fe5e6bf89 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 03:04:42.031741 26357 raft_consensus.cc:686] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 LEADER]: Becoming Leader. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:04:42.039646 26337 raft_consensus.cc:3037] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:41.994249 26370 raft_consensus.cc:504] T 06f4a7eaf6534e4986c9fd35911038e4 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:04:42.037846 26377 consensus_meta.cc:287] T edcf6e6a422d4d198d4fbb0f7d3d845d P 869b0a8575b54d08ae2817fad5dd42d5: Time spent flushing consensus metadata: real 0.511s user 0.008s sys 0.004s
I20260925 03:04:41.966055 26414 consensus_queue.cc:1035] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.077s
I20260925 03:04:42.103000 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:42.029838 26262 consensus_queue.cc:1035] T 100c113a9fd84cc8adf2ec7b33ea0093 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.071s
I20260925 03:04:42.116034 24471 raft_consensus.cc:2379] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c in current term 5: Already voted for candidate 1bc1f2e62ab44af995165bd5116a9d43 in this term.
W20260925 03:04:42.116453 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 67286.1us
I20260925 03:04:42.083730 26241 raft_consensus.cc:2781] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260925 03:04:42.038450 26365 raft_consensus.cc:2726] T 3d89396576884f7c8901dba54f783947 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260925 03:04:42.131325 26422 raft_consensus.cc:2726] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260925 03:04:42.103511 26293 raft_consensus.cc:2726] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260925 03:04:42.102321 26363 raft_consensus.cc:482] T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:42.106545 26417 raft_consensus.cc:2781] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Leader election won for term 4
W20260925 03:04:42.141759 24436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 169077us
W20260925 03:04:42.129526 26325 consensus_meta.cc:287] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5: Time spent flushing consensus metadata: real 0.504s user 0.010s sys 0.000s
W20260925 03:04:42.148574 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 121631us
W20260925 03:04:42.148633 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 243737us
W20260925 03:04:42.155889 25165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 123658us
I20260925 03:04:42.129069 26241 raft_consensus.cc:482] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:42.135281 26371 raft_consensus.cc:686] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 LEADER]: Becoming Leader. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:04:42.157922 24514 leader_election.cc:302] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:42.162282 24437 leader_election.cc:302] T b4ebd2b043024a548005c8b6b00b0449 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: 1bc1f2e62ab44af995165bd5116a9d43, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:42.165496 23300 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260925 03:04:42.142762 26379 raft_consensus.cc:482] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:42.168059 25165 leader_election.cc:302] T b1e85248d3b04b979dd080caeeb02fdb P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:04:42.142380 26285 leader_election.cc:288] T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:42.149371 26209 leader_election.cc:288] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
W20260925 03:04:42.171378 26420 log.cc:924] Time spent T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.056s user 0.001s sys 0.000s
W20260925 03:04:42.176573 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 59163.6us
I20260925 03:04:42.164597 26241 raft_consensus.cc:3037] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:42.180876 24436 leader_election.cc:302] T 0be4e0aa984f4f2b81a58e382e434ad5 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:42.182374 24514 leader_election.cc:302] T 48ca6ccd98194cf2af7b5278d15bf4ea P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:04:42.159093 26363 raft_consensus.cc:504] T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:42.174626 26381 raft_consensus.cc:2726] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260925 03:04:42.179257 26422 raft_consensus.cc:2726] T b4ebd2b043024a548005c8b6b00b0449 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260925 03:04:42.208634 26416 log.cc:924] Time spent T 100c113a9fd84cc8adf2ec7b33ea0093 P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.210s user 0.000s sys 0.001s
W20260925 03:04:42.209403 26407 log.cc:924] Time spent T 550118bc1a0641ce9803bdbf43f2069f P 869b0a8575b54d08ae2817fad5dd42d5: Append to log took a long time: real 0.199s user 0.001s sys 0.001s
I20260925 03:04:42.211282 24437 leader_election.cc:302] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:04:42.216995 24514 leader_election.cc:302] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:42.178282 26379 raft_consensus.cc:504] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:42.195057 26368 leader_election.cc:288] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:42.196095 26375 raft_consensus.cc:2781] T 48ca6ccd98194cf2af7b5278d15bf4ea P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260925 03:04:42.210085 26415 log.cc:924] Time spent T 00000000000000000000000000000000 P 2d92cc0732c4490eb7d98d353826cce1: Append to log took a long time: real 0.122s user 0.002s sys 0.000s
I20260925 03:04:42.288267 26422 raft_consensus.cc:2781] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260925 03:04:42.175559 26357 consensus_queue.cc:237] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:42.290153 24437 leader_election.cc:302] T f22ddd7d1c444dc58f45a1fb0b3e02bc P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:42.297544 24436 leader_election.cc:302] T b627ecb4a5384ef392aa695e06599c39 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
W20260925 03:04:42.299284 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 82389us
I20260925 03:04:42.225056 25979 raft_consensus.cc:482] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:42.294476 26362 raft_consensus.cc:2726] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260925 03:04:42.301898 24514 leader_election.cc:302] T 367277daf8d04778b3fa4ed94d71580d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:42.302245 26363 leader_election.cc:288] T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:42.302778 26363 raft_consensus.cc:2726] T 367277daf8d04778b3fa4ed94d71580d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260925 03:04:42.303640 24514 leader_election.cc:302] T edcf6e6a422d4d198d4fbb0f7d3d845d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:42.304076 26439 raft_consensus.cc:2781] T b1e85248d3b04b979dd080caeeb02fdb P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Leader election won for term 4
I20260925 03:04:42.304593 26439 raft_consensus.cc:686] T b1e85248d3b04b979dd080caeeb02fdb P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 LEADER]: Becoming Leader. State: Replica: 1bc1f2e62ab44af995165bd5116a9d43, State: Running, Role: LEADER
I20260925 03:04:42.305008 26363 raft_consensus.cc:2726] T edcf6e6a422d4d198d4fbb0f7d3d845d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260925 03:04:42.305756 24514 leader_election.cc:302] T cc2df40db08649c9a75e90000b878559 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:42.305490 26439 consensus_queue.cc:237] T b1e85248d3b04b979dd080caeeb02fdb P 1bc1f2e62ab44af995165bd5116a9d43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:42.307271 24514 leader_election.cc:398] T a7aa47c48404465b94a36fd819fe3c7a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187) with higher term. Message: Invalid argument: T a7aa47c48404465b94a36fd819fe3c7a P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 2. Current term is 3.
I20260925 03:04:42.307682 26363 raft_consensus.cc:2726] T cc2df40db08649c9a75e90000b878559 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260925 03:04:42.308349 24514 leader_election.cc:401] T a7aa47c48404465b94a36fd819fe3c7a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260925 03:04:42.308971 24437 leader_election.cc:302] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:42.309557 26363 raft_consensus.cc:2726] T a7aa47c48404465b94a36fd819fe3c7a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187) with higher term. Message: Invalid argument: T a7aa47c48404465b94a36fd819fe3c7a P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 2. Current term is 3.
I20260925 03:04:42.309724 26325 raft_consensus.cc:2726] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260925 03:04:42.310138 24437 leader_election.cc:302] T 8e5819c15eac4544a57e5ee6c9d4f117 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:04:42.310832 26325 raft_consensus.cc:2781] T 8e5819c15eac4544a57e5ee6c9d4f117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260925 03:04:42.311203 26325 raft_consensus.cc:482] T 8e5819c15eac4544a57e5ee6c9d4f117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:42.311259 24437 leader_election.cc:302] T 25102ba08e4e49388f6176badcd10a25 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:42.311612 26325 raft_consensus.cc:3037] T 8e5819c15eac4544a57e5ee6c9d4f117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:42.312420 26425 raft_consensus.cc:2726] T f22ddd7d1c444dc58f45a1fb0b3e02bc P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260925 03:04:42.216500 26371 consensus_queue.cc:237] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:42.318836 24465 raft_consensus.cc:1261] T cc2df40db08649c9a75e90000b878559 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
I20260925 03:04:42.319401 24465 pending_rounds.cc:76] T cc2df40db08649c9a75e90000b878559 P 869b0a8575b54d08ae2817fad5dd42d5: Aborting all ops after (but not including) 1
I20260925 03:04:42.319096 26325 raft_consensus.cc:504] T 8e5819c15eac4544a57e5ee6c9d4f117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:42.319651 24465 pending_rounds.cc:97] T cc2df40db08649c9a75e90000b878559 P 869b0a8575b54d08ae2817fad5dd42d5: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20260925 03:04:42.319983 24465 raft_consensus.cc:2866] T cc2df40db08649c9a75e90000b878559 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260925 03:04:42.227520 26436 consensus_queue.cc:1046] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.138s
I20260925 03:04:42.321539 24514 leader_election.cc:398] T 8e5819c15eac4544a57e5ee6c9d4f117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187) with higher term. Message: Invalid argument: T 8e5819c15eac4544a57e5ee6c9d4f117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 2. Current term is 3.
I20260925 03:04:42.321496 26436 raft_consensus.cc:3032] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 LEADER]: Stepping down as leader of term 5
I20260925 03:04:42.321982 26436 raft_consensus.cc:729] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:04:42.322716 26436 consensus_queue.cc:260] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:42.290397 26377 consensus_queue.cc:1046] T 550118bc1a0641ce9803bdbf43f2069f P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.034s
I20260925 03:04:42.323902 26436 raft_consensus.cc:3037] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:04:42.227389 26375 raft_consensus.cc:482] T 48ca6ccd98194cf2af7b5278d15bf4ea P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:42.326267 24514 leader_election.cc:401] T 8e5819c15eac4544a57e5ee6c9d4f117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260925 03:04:42.328809 26337 raft_consensus.cc:2726] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260925 03:04:42.330317 26379 leader_election.cc:288] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:42.294301 26386 consensus_queue.cc:1046] T 0b5df303cd6a4d59aca5c18bbbff24ed P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.006s
I20260925 03:04:42.294061 26422 raft_consensus.cc:482] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
W20260925 03:04:42.290110 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 90833us
I20260925 03:04:42.345959 26406 raft_consensus.cc:2726] T 8e5819c15eac4544a57e5ee6c9d4f117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187) with higher term. Message: Invalid argument: T 8e5819c15eac4544a57e5ee6c9d4f117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 2. Current term is 3.
I20260925 03:04:42.346414 26409 raft_consensus.cc:504] T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:42.346415 26432 raft_consensus.cc:482] T 7993f9c3e2a44f408bbf5d1e395e2306 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:42.348680 26427 raft_consensus.cc:3032] T 25102ba08e4e49388f6176badcd10a25 P eeab93abfdb141a895002ec9f312296c [term 1 LEADER]: Stepping down as leader of term 1
I20260925 03:04:42.349277 26377 raft_consensus.cc:2781] T b627ecb4a5384ef392aa695e06599c39 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader election won for term 5
I20260925 03:04:42.349502 24437 leader_election.cc:302] T 43366a0a2f9d4344bf4e37ffd290ef8a P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:42.353025 24436 leader_election.cc:302] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:42.325284 26370 leader_election.cc:288] T 06f4a7eaf6534e4986c9fd35911038e4 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
W20260925 03:04:42.357228 25165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 189254us
I20260925 03:04:42.330729 26375 raft_consensus.cc:3037] T 48ca6ccd98194cf2af7b5278d15bf4ea P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Advancing to term 3
W20260925 03:04:42.375924 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 54513.1us
I20260925 03:04:42.391157 26248 raft_consensus.cc:686] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [term 6 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:42.393273 26248 consensus_queue.cc:237] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:42.350028 26337 raft_consensus.cc:482] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260925 03:04:42.401836 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 52434.7us
I20260925 03:04:42.352994 26369 raft_consensus.cc:3032] T 550118bc1a0641ce9803bdbf43f2069f P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 LEADER]: Stepping down as leader of term 3
I20260925 03:04:42.351329 26365 raft_consensus.cc:3037] T 25102ba08e4e49388f6176badcd10a25 P 869b0a8575b54d08ae2817fad5dd42d5 [term 1 FOLLOWER]: Advancing to term 2
W20260925 03:04:42.416939 24436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 64004.1us
I20260925 03:04:42.351327 26422 raft_consensus.cc:3037] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:04:42.375115 26086 raft_consensus.cc:482] T 550118bc1a0641ce9803bdbf43f2069f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:42.425864 24437 leader_election.cc:302] T 7993f9c3e2a44f408bbf5d1e395e2306 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:04:42.373870 26427 raft_consensus.cc:729] T 25102ba08e4e49388f6176badcd10a25 P eeab93abfdb141a895002ec9f312296c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:42.325459 25979 raft_consensus.cc:504] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:42.452530 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b5df303cd6a4d59aca5c18bbbff24ed" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
W20260925 03:04:42.457633 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 427336us
I20260925 03:04:42.424208 24514 leader_election.cc:302] T 1414dcfe53f64a78ae413531eb8ec117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:42.467003 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15861cd28e354598a3c6bfc4337c34bb" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:42.418633 26370 raft_consensus.cc:2726] T 43366a0a2f9d4344bf4e37ffd290ef8a P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260925 03:04:42.374696 26379 consensus_queue.cc:1035] T 442d4a29b1d743599f4b6b4871e26668 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.051s
I20260925 03:04:42.420261 26369 raft_consensus.cc:729] T 550118bc1a0641ce9803bdbf43f2069f P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:04:42.478063 24473 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0be4e0aa984f4f2b81a58e382e434ad5"
dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
mode: GRACEFUL
new_leader_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
from {username='slave'} at 127.0.0.1:38832
I20260925 03:04:42.480171 24465 raft_consensus.cc:1261] T 4e5332a5d329493991dff994e70ea92c P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20260925 03:04:42.481002 24465 pending_rounds.cc:76] T 4e5332a5d329493991dff994e70ea92c P 869b0a8575b54d08ae2817fad5dd42d5: Aborting all ops after (but not including) 2
I20260925 03:04:42.481256 24465 pending_rounds.cc:97] T 4e5332a5d329493991dff994e70ea92c P 869b0a8575b54d08ae2817fad5dd42d5: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20260925 03:04:42.481527 24465 raft_consensus.cc:2866] T 4e5332a5d329493991dff994e70ea92c P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260925 03:04:42.427785 25193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43"
I20260925 03:04:42.452534 26400 raft_consensus.cc:3037] T e6b6ec010caf4689b8c90449764f5cf9 P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:42.488512 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c23337a7cb74184a0e57b7095cb6a11" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:42.459797 26362 raft_consensus.cc:482] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:42.452889 26386 raft_consensus.cc:482] T a7aa47c48404465b94a36fd819fe3c7a P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:42.493137 26438 raft_consensus.cc:2781] T 0be4e0aa984f4f2b81a58e382e434ad5 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader election won for term 4
I20260925 03:04:42.501492 26438 raft_consensus.cc:686] T 0be4e0aa984f4f2b81a58e382e434ad5 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 LEADER]: Becoming Leader. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:04:42.502415 26438 consensus_queue.cc:237] T 0be4e0aa984f4f2b81a58e382e434ad5 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:42.503280 26369 consensus_queue.cc:260] T 550118bc1a0641ce9803bdbf43f2069f P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:42.428378 26377 raft_consensus.cc:686] T b627ecb4a5384ef392aa695e06599c39 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 LEADER]: Becoming Leader. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:04:42.504640 26413 raft_consensus.cc:482] T b1e85248d3b04b979dd080caeeb02fdb P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260925 03:04:42.505331 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 88849.8us
I20260925 03:04:42.505271 26413 raft_consensus.cc:504] T b1e85248d3b04b979dd080caeeb02fdb P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:42.506206 24437 leader_election.cc:302] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
W20260925 03:04:42.487311 26428 log.cc:924] Time spent T 21c43af5e6844fdaacc8c625fa01fd8d P 869b0a8575b54d08ae2817fad5dd42d5: Append to log took a long time: real 0.058s user 0.000s sys 0.000s
W20260925 03:04:42.487313 26443 log.cc:924] Time spent T c85f5d1d3f004cf5817bb594fd8e6469 P 869b0a8575b54d08ae2817fad5dd42d5: Append to log took a long time: real 0.059s user 0.000s sys 0.000s
I20260925 03:04:42.489341 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b627ecb4a5384ef392aa695e06599c39" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:42.414883 26367 consensus_queue.cc:1035] T cc2df40db08649c9a75e90000b878559 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.065s
I20260925 03:04:42.465291 26417 raft_consensus.cc:686] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 LEADER]: Becoming Leader. State: Replica: 1bc1f2e62ab44af995165bd5116a9d43, State: Running, Role: LEADER
W20260925 03:04:42.486112 26373 consensus_meta.cc:287] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent flushing consensus metadata: real 0.633s user 0.009s sys 0.000s
I20260925 03:04:42.416244 26337 raft_consensus.cc:504] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:42.478026 26406 raft_consensus.cc:2726] T 1414dcfe53f64a78ae413531eb8ec117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260925 03:04:42.509768 24621 raft_consensus.cc:2379] T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 5: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:04:42.495359 25193 raft_consensus.cc:2379] T 29637f5a4a504ce6ba29033ccd0cd716 P 1bc1f2e62ab44af995165bd5116a9d43 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 in current term 5: Already voted for candidate eeab93abfdb141a895002ec9f312296c in this term.
W20260925 03:04:42.514607 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 110226us
I20260925 03:04:42.491803 26371 raft_consensus.cc:2781] T 7993f9c3e2a44f408bbf5d1e395e2306 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader election won for term 4
I20260925 03:04:42.488309 25196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c81a1b3b616492ab6c964f45af36077" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43"
I20260925 03:04:42.518235 26375 raft_consensus.cc:504] T 48ca6ccd98194cf2af7b5278d15bf4ea P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:04:42.519876 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 47059.8us
I20260925 03:04:42.521385 26437 raft_consensus.cc:2781] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader election won for term 4
I20260925 03:04:42.522014 26436 raft_consensus.cc:2726] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260925 03:04:42.505587 26377 consensus_queue.cc:237] T b627ecb4a5384ef392aa695e06599c39 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:42.495469 26429 raft_consensus.cc:482] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:42.523973 26293 raft_consensus.cc:3032] T 0b5df303cd6a4d59aca5c18bbbff24ed P eeab93abfdb141a895002ec9f312296c [term 4 LEADER]: Stepping down as leader of term 4
I20260925 03:04:42.387089 26432 raft_consensus.cc:504] T 7993f9c3e2a44f408bbf5d1e395e2306 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:42.500804 26421 raft_consensus.cc:482] T 37a157996e944db2906110d87ecb0c54 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:42.469614 26086 raft_consensus.cc:504] T 550118bc1a0641ce9803bdbf43f2069f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:04:42.533941 26442 log.cc:924] Time spent T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5: Append to log took a long time: real 0.053s user 0.000s sys 0.000s
I20260925 03:04:42.534693 26375 leader_election.cc:288] T 48ca6ccd98194cf2af7b5278d15bf4ea P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
W20260925 03:04:42.534808 26445 log.cc:924] Time spent T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.059s user 0.000s sys 0.000s
I20260925 03:04:42.537484 24514 leader_election.cc:302] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
W20260925 03:04:42.523710 24585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.4:43979, user_credentials={real_user=slave}} blocked reactor thread for 100221us
I20260925 03:04:42.540675 24436 leader_election.cc:302] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: 1bc1f2e62ab44af995165bd5116a9d43, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:42.522994 25194 raft_consensus.cc:2454] T 71f461a6e2ff434993b78a2d077f0abd P 1bc1f2e62ab44af995165bd5116a9d43 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 5.
I20260925 03:04:42.519449 26366 raft_consensus.cc:482] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:42.496548 25191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:42.559518 26366 raft_consensus.cc:504] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:42.561556 26366 leader_election.cc:288] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:42.562058 26441 raft_consensus.cc:482] T 21c43af5e6844fdaacc8c625fa01fd8d P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:42.562533 26441 raft_consensus.cc:504] T 21c43af5e6844fdaacc8c625fa01fd8d P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:42.565068 25198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:42.565455 25979 leader_election.cc:288] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:42.529671 26369 raft_consensus.cc:3037] T 550118bc1a0641ce9803bdbf43f2069f P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:42.559724 25191 raft_consensus.cc:2454] T 29637f5a4a504ce6ba29033ccd0cd716 P 1bc1f2e62ab44af995165bd5116a9d43 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 5.
I20260925 03:04:42.558408 26431 consensus_queue.cc:1035] T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.032s
I20260925 03:04:42.544432 26371 raft_consensus.cc:686] T 7993f9c3e2a44f408bbf5d1e395e2306 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 LEADER]: Becoming Leader. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:04:42.577409 26371 consensus_queue.cc:237] T 7993f9c3e2a44f408bbf5d1e395e2306 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:42.579473 24615 raft_consensus.cc:1261] T 21c43af5e6844fdaacc8c625fa01fd8d P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260925 03:04:42.579952 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e5819c15eac4544a57e5ee6c9d4f117" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
W20260925 03:04:42.581434 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 48291.4us
I20260925 03:04:42.495286 26386 raft_consensus.cc:504] T a7aa47c48404465b94a36fd819fe3c7a P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:42.566056 26426 raft_consensus.cc:2726] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260925 03:04:42.564548 26441 leader_election.cc:288] T 21c43af5e6844fdaacc8c625fa01fd8d P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:42.548067 26293 raft_consensus.cc:729] T 0b5df303cd6a4d59aca5c18bbbff24ed P eeab93abfdb141a895002ec9f312296c [term 4 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
W20260925 03:04:42.587276 25163 leader_election.cc:334] T 7e49248643df4131a1c42e74244e05c2 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389): Timed out: RequestConsensusVote RPC to 127.22.193.2:36389 timed out after 1.831s (SENT)
I20260925 03:04:42.590309 24465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60d45b7d467d4e1ab92367f6d7d52b15" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:42.600199 25163 leader_election.cc:302] T 7e49248643df4131a1c42e74244e05c2 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:42.600716 24465 raft_consensus.cc:2396] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260925 03:04:42.591224 24467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21c43af5e6844fdaacc8c625fa01fd8d" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:42.593940 25191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ac466793d094325ace224708b2158ed" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:42.594506 24469 raft_consensus.cc:1226] T 8e5819c15eac4544a57e5ee6c9d4f117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 3. Current term is 4. Ops: []
I20260925 03:04:42.527482 26373 raft_consensus.cc:504] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:42.557407 26437 raft_consensus.cc:686] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 LEADER]: Becoming Leader. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:04:42.613054 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:42.575621 25198 raft_consensus.cc:2379] T 7e49248643df4131a1c42e74244e05c2 P 1bc1f2e62ab44af995165bd5116a9d43 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c in current term 5: Already voted for candidate 1bc1f2e62ab44af995165bd5116a9d43 in this term.
I20260925 03:04:42.605917 24615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48ca6ccd98194cf2af7b5278d15bf4ea" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:42.607228 26373 leader_election.cc:288] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:42.609980 26432 leader_election.cc:288] T 7993f9c3e2a44f408bbf5d1e395e2306 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:42.568763 26325 leader_election.cc:288] T 8e5819c15eac4544a57e5ee6c9d4f117 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:42.610634 26422 raft_consensus.cc:504] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:42.611071 26413 leader_election.cc:288] T b1e85248d3b04b979dd080caeeb02fdb P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:42.622367 24585 leader_election.cc:302] T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:04:42.514118 26362 raft_consensus.cc:504] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:42.613724 24617 raft_consensus.cc:2454] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 5.
I20260925 03:04:42.630676 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06f4a7eaf6534e4986c9fd35911038e4" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:42.582211 26375 raft_consensus.cc:2726] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260925 03:04:42.604449 26409 leader_election.cc:288] T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:42.581579 26270 consensus_queue.cc:1035] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.032s
I20260925 03:04:42.637629 25198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1e85248d3b04b979dd080caeeb02fdb" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:42.496646 26427 consensus_queue.cc:260] T 25102ba08e4e49388f6176badcd10a25 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:04:42.639210 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 101794us
I20260925 03:04:42.639952 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e54ca1757ee4e42956a9cebbeca4f62" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
W20260925 03:04:42.647444 25163 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 60214.2us
I20260925 03:04:42.601037 26374 consensus_queue.cc:1035] T 21c43af5e6844fdaacc8c625fa01fd8d P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.040s
I20260925 03:04:42.635802 26439 raft_consensus.cc:2726] T 7e49248643df4131a1c42e74244e05c2 P 1bc1f2e62ab44af995165bd5116a9d43 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260925 03:04:42.650070 24586 leader_election.cc:302] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters: 1bc1f2e62ab44af995165bd5116a9d43
I20260925 03:04:42.612543 26241 raft_consensus.cc:504] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:42.651719 24464 raft_consensus.cc:2454] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 6.
I20260925 03:04:42.550163 26429 raft_consensus.cc:504] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:42.553601 26421 raft_consensus.cc:504] T 37a157996e944db2906110d87ecb0c54 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:42.713716 24472 raft_consensus.cc:1226] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 5. Current term is 6. Ops: []
I20260925 03:04:42.715317 25165 leader_election.cc:302] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5; no voters:
W20260925 03:04:42.751143 24585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.4:43979, user_credentials={real_user=slave}} blocked reactor thread for 128882us
I20260925 03:04:42.613312 26293 consensus_queue.cc:260] T 0b5df303cd6a4d59aca5c18bbbff24ed P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:04:42.773114 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 123129us
W20260925 03:04:42.773784 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 113283us
I20260925 03:04:42.648989 26337 leader_election.cc:288] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:42.601181 26440 consensus_queue.cc:1035] T ac9357d144aa49789faaf31fe5e6bf89 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.048s
W20260925 03:04:42.785418 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 237190us
I20260925 03:04:42.635003 26417 consensus_queue.cc:237] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:42.781947 24436 leader_election.cc:302] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
W20260925 03:04:42.782485 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 131151us
I20260925 03:04:42.760385 26439 raft_consensus.cc:2781] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260925 03:04:42.714074 26375 raft_consensus.cc:482] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:42.776785 26366 raft_consensus.cc:2781] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader election won for term 4
I20260925 03:04:42.780328 26367 raft_consensus.cc:3037] T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:42.795194 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48ca6ccd98194cf2af7b5278d15bf4ea" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:42.798960 24437 leader_election.cc:302] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:04:42.800879 26436 raft_consensus.cc:2781] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 03:04:42.801436 26436 raft_consensus.cc:686] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 LEADER]: Becoming Leader. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:04:42.793725 26270 consensus_queue.cc:1046] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.007s
I20260925 03:04:42.801627 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7993f9c3e2a44f408bbf5d1e395e2306" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:42.802048 24437 leader_election.cc:302] T e6b6ec010caf4689b8c90449764f5cf9 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 1bc1f2e62ab44af995165bd5116a9d43
I20260925 03:04:42.802840 26426 raft_consensus.cc:2781] T e6b6ec010caf4689b8c90449764f5cf9 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader election won for term 4
I20260925 03:04:42.803328 26426 raft_consensus.cc:686] T e6b6ec010caf4689b8c90449764f5cf9 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 LEADER]: Becoming Leader. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:04:42.803727 26422 leader_election.cc:288] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:42.794973 24514 leader_election.cc:398] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 pre-election: Vote denied by peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187) with higher term. Message: Invalid argument: T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 4. Current term is 5.
I20260925 03:04:42.810220 24467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1e85248d3b04b979dd080caeeb02fdb" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:42.613406 26437 consensus_queue.cc:237] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:42.815769 26429 leader_election.cc:288] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:42.817272 24542 raft_consensus.cc:1261] T c85f5d1d3f004cf5817bb594fd8e6469 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260925 03:04:42.818858 26409 consensus_queue.cc:1035] T 442d4a29b1d743599f4b6b4871e26668 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260925 03:04:42.825676 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "550118bc1a0641ce9803bdbf43f2069f" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:42.827793 24542 raft_consensus.cc:1261] T cc2df40db08649c9a75e90000b878559 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260925 03:04:42.810938 24467 raft_consensus.cc:2454] T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 4.
I20260925 03:04:42.812630 26293 raft_consensus.cc:3037] T 0b5df303cd6a4d59aca5c18bbbff24ed P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:42.819635 26366 raft_consensus.cc:686] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [term 4 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:42.820663 26383 raft_consensus.cc:3032] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c [term 3 LEADER]: Stepping down as leader of term 3
I20260925 03:04:42.821165 26436 consensus_queue.cc:237] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:42.824296 26421 leader_election.cc:288] T 37a157996e944db2906110d87ecb0c54 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:42.830555 24465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5aec38e66a947c681e6c2a72b05e6ca" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:42.833372 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "550118bc1a0641ce9803bdbf43f2069f" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:42.835570 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:42.836326 24615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7691c6b475428c82d6181d693d26f3" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:42.879500 24546 raft_consensus.cc:3037] T 550118bc1a0641ce9803bdbf43f2069f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:42.845173 24619 raft_consensus.cc:2379] T 550118bc1a0641ce9803bdbf43f2069f P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 4: Already voted for candidate eeab93abfdb141a895002ec9f312296c in this term.
I20260925 03:04:42.880302 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e725b82985746b39c0cc8f671c95353" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:42.792924 26325 raft_consensus.cc:2781] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260925 03:04:42.789464 26439 raft_consensus.cc:482] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:42.846071 24437 leader_election.cc:302] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:04:42.868592 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e54ca1757ee4e42956a9cebbeca4f62" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:42.868952 24540 raft_consensus.cc:1226] T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 3. Current term is 4. Ops: []
I20260925 03:04:42.868615 25198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5aec38e66a947c681e6c2a72b05e6ca" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:42.886121 24437 leader_election.cc:302] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:04:42.874138 24588 leader_election.cc:302] T b1e85248d3b04b979dd080caeeb02fdb P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters: 1bc1f2e62ab44af995165bd5116a9d43
I20260925 03:04:42.809016 26086 leader_election.cc:288] T 550118bc1a0641ce9803bdbf43f2069f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:42.880859 24544 raft_consensus.cc:1261] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260925 03:04:42.878916 24465 raft_consensus.cc:2454] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 6.
I20260925 03:04:42.881983 24467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7aa47c48404465b94a36fd819fe3c7a" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:42.908586 24467 raft_consensus.cc:2454] T a7aa47c48404465b94a36fd819fe3c7a P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 3.
I20260925 03:04:42.883391 24548 raft_consensus.cc:2379] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c in current term 5: Already voted for candidate 1bc1f2e62ab44af995165bd5116a9d43 in this term.
I20260925 03:04:42.908993 24615 raft_consensus.cc:1261] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260925 03:04:42.884716 24614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37a157996e944db2906110d87ecb0c54" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:42.887262 24437 leader_election.cc:302] T a7aa47c48404465b94a36fd819fe3c7a P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:04:42.872819 26371 raft_consensus.cc:482] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:42.805869 26375 raft_consensus.cc:504] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:42.887691 26325 raft_consensus.cc:482] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:42.870282 26383 raft_consensus.cc:729] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:42.889619 24617 raft_consensus.cc:2454] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 6.
I20260925 03:04:42.886531 25198 raft_consensus.cc:2454] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 6.
I20260925 03:04:42.887832 26386 leader_election.cc:288] T a7aa47c48404465b94a36fd819fe3c7a P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:42.913159 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ac466793d094325ace224708b2158ed" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:42.929039 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7993f9c3e2a44f408bbf5d1e395e2306" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:42.929759 24548 raft_consensus.cc:2454] T 7993f9c3e2a44f408bbf5d1e395e2306 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 4.
I20260925 03:04:42.889230 26439 raft_consensus.cc:3037] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:04:42.915670 24588 leader_election.cc:302] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:42.866590 26285 consensus_queue.cc:1046] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.036s
I20260925 03:04:42.905165 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60d45b7d467d4e1ab92367f6d7d52b15" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:42.889385 26421 raft_consensus.cc:2781] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader election won for term 5
I20260925 03:04:42.804258 26426 consensus_queue.cc:237] T e6b6ec010caf4689b8c90449764f5cf9 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:42.912070 24473 raft_consensus.cc:595] T 0be4e0aa984f4f2b81a58e382e434ad5 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 LEADER]: Received request to transfer leadership to TS a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:42.891086 26379 raft_consensus.cc:2781] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260925 03:04:42.926291 24546 raft_consensus.cc:2454] T 550118bc1a0641ce9803bdbf43f2069f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 4.
I20260925 03:04:42.927086 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21c43af5e6844fdaacc8c625fa01fd8d" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:42.906966 26425 raft_consensus.cc:482] T 9e8af2641e124f9589c5ccfdac785f63 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:42.932070 26366 consensus_queue.cc:237] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:42.936275 24540 raft_consensus.cc:2454] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 5.
W20260925 03:04:42.943459 26454 log.cc:924] Time spent T b1e85248d3b04b979dd080caeeb02fdb P 1bc1f2e62ab44af995165bd5116a9d43: Append to log took a long time: real 0.438s user 0.004s sys 0.004s
I20260925 03:04:42.945735 26375 leader_election.cc:288] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:42.918594 26413 raft_consensus.cc:2781] T b1e85248d3b04b979dd080caeeb02fdb P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260925 03:04:42.952026 24544 raft_consensus.cc:2396] T 21c43af5e6844fdaacc8c625fa01fd8d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260925 03:04:42.950595 24588 leader_election.cc:302] T a7aa47c48404465b94a36fd819fe3c7a P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
W20260925 03:04:42.955986 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 161146us
I20260925 03:04:42.924536 26325 raft_consensus.cc:3037] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:04:42.922605 26409 raft_consensus.cc:2781] T a7aa47c48404465b94a36fd819fe3c7a P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 03:04:42.921308 26370 consensus_queue.cc:1035] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.049s
I20260925 03:04:42.928009 26362 leader_election.cc:288] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:42.947100 24541 raft_consensus.cc:1261] T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260925 03:04:43.018257 24514 leader_election.cc:398] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187) with higher term. Message: Invalid argument: T 7b8e0d260a884ecdb20f79de46c36b19 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 3. Current term is 4.
W20260925 03:04:43.019335 24585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.4:43979, user_credentials={real_user=slave}} blocked reactor thread for 71230.8us
I20260925 03:04:42.960357 25198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "293ef69caf514ac98510a68495f54eda" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:42.918860 26371 raft_consensus.cc:504] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:43.042266 24540 raft_consensus.cc:1261] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260925 03:04:42.957422 26379 raft_consensus.cc:482] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:42.956372 26373 raft_consensus.cc:482] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:42.969604 26367 raft_consensus.cc:2726] T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260925 03:04:42.970001 26241 leader_election.cc:288] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:42.978016 25979 raft_consensus.cc:2781] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260925 03:04:42.975136 26413 raft_consensus.cc:482] T b1e85248d3b04b979dd080caeeb02fdb P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260925 03:04:43.054414 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 103905us
I20260925 03:04:43.040978 25942 raft_consensus.cc:3032] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [term 5 LEADER]: Stepping down as leader of term 5
W20260925 03:04:43.020953 26442 log.cc:924] Time spent T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5: Append to log took a long time: real 0.071s user 0.000s sys 0.000s
I20260925 03:04:43.087450 24540 raft_consensus.cc:1226] T b030d84c670b48d5bff337ccc0c77626 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 5. Current term is 6. Ops: []
I20260925 03:04:43.066146 25942 raft_consensus.cc:729] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [term 5 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:43.049870 26406 raft_consensus.cc:482] T c85f5d1d3f004cf5817bb594fd8e6469 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:43.050177 26379 raft_consensus.cc:3037] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Advancing to term 4
W20260925 03:04:43.158157 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 99266.9us
I20260925 03:04:43.056715 25979 raft_consensus.cc:482] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:43.060051 26413 raft_consensus.cc:3037] T b1e85248d3b04b979dd080caeeb02fdb P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:43.059803 26247 raft_consensus.cc:2781] T a7aa47c48404465b94a36fd819fe3c7a P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260925 03:04:43.089138 26365 consensus_meta.cc:287] T 25102ba08e4e49388f6176badcd10a25 P 869b0a8575b54d08ae2817fad5dd42d5: Time spent flushing consensus metadata: real 0.635s user 0.010s sys 0.000s
I20260925 03:04:43.166376 24514 leader_election.cc:302] T c88cbd39e01f44628a094af516be2833 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:43.169358 26264 raft_consensus.cc:482] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260925 03:04:43.173612 26453 log.cc:924] Time spent T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5: Append to log took a long time: real 0.422s user 0.003s sys 0.009s
I20260925 03:04:43.173908 26450 raft_consensus.cc:482] T a4abe0ea757e4886be4945c5001e529a P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:43.174530 25942 consensus_queue.cc:260] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:43.178022 24466 raft_consensus.cc:1226] T 06f4a7eaf6534e4986c9fd35911038e4 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 4. Current term is 5. Ops: []
I20260925 03:04:43.178470 26448 raft_consensus.cc:482] T edcf6e6a422d4d198d4fbb0f7d3d845d P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:43.179453 24465 raft_consensus.cc:1261] T 37a157996e944db2906110d87ecb0c54 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20260925 03:04:43.205158 24465 pending_rounds.cc:76] T 37a157996e944db2906110d87ecb0c54 P 869b0a8575b54d08ae2817fad5dd42d5: Aborting all ops after (but not including) 2
I20260925 03:04:43.205394 24465 pending_rounds.cc:97] T 37a157996e944db2906110d87ecb0c54 P 869b0a8575b54d08ae2817fad5dd42d5: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20260925 03:04:43.205618 24465 raft_consensus.cc:2866] T 37a157996e944db2906110d87ecb0c54 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260925 03:04:43.179780 26429 consensus_queue.cc:1035] T c85f5d1d3f004cf5817bb594fd8e6469 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20260925 03:04:43.181252 26467 log.cc:924] Time spent T 442d4a29b1d743599f4b6b4871e26668 P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.245s user 0.000s sys 0.000s
W20260925 03:04:43.181876 26466 log.cc:924] Time spent T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.299s user 0.001s sys 0.000s
I20260925 03:04:43.183071 26371 leader_election.cc:288] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
W20260925 03:04:43.185732 26468 log.cc:924] Time spent T 0be4e0aa984f4f2b81a58e382e434ad5 P 869b0a8575b54d08ae2817fad5dd42d5: Append to log took a long time: real 0.142s user 0.000s sys 0.000s
W20260925 03:04:43.187469 26461 log.cc:924] Time spent T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.340s user 0.001s sys 0.000s
I20260925 03:04:43.060472 26373 raft_consensus.cc:504] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:04:43.188481 26369 consensus_meta.cc:287] T 550118bc1a0641ce9803bdbf43f2069f P 869b0a8575b54d08ae2817fad5dd42d5: Time spent flushing consensus metadata: real 0.615s user 0.008s sys 0.000s
I20260925 03:04:43.126824 26400 raft_consensus.cc:482] T e6b6ec010caf4689b8c90449764f5cf9 P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:43.200672 26264 raft_consensus.cc:504] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:43.165848 25979 raft_consensus.cc:3037] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:04:43.211424 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7aa47c48404465b94a36fd819fe3c7a" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:43.197441 26247 raft_consensus.cc:482] T a7aa47c48404465b94a36fd819fe3c7a P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:42.977458 26425 raft_consensus.cc:504] T 9e8af2641e124f9589c5ccfdac785f63 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:43.204777 26365 raft_consensus.cc:2726] T 25102ba08e4e49388f6176badcd10a25 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260925 03:04:43.219550 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 92450.7us
I20260925 03:04:43.228852 24514 leader_election.cc:302] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:04:43.224282 24544 raft_consensus.cc:2454] T a7aa47c48404465b94a36fd819fe3c7a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 3.
I20260925 03:04:43.021561 26383 consensus_queue.cc:260] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:43.161991 26406 raft_consensus.cc:504] T c85f5d1d3f004cf5817bb594fd8e6469 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:04:43.234795 26473 log.cc:924] Time spent T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43: Append to log took a long time: real 0.294s user 0.001s sys 0.000s
I20260925 03:04:43.238711 24541 raft_consensus.cc:1226] T 48ca6ccd98194cf2af7b5278d15bf4ea P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 2. Current term is 3. Ops: []
I20260925 03:04:43.223973 26400 raft_consensus.cc:504] T e6b6ec010caf4689b8c90449764f5cf9 P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:43.241981 26400 leader_election.cc:288] T e6b6ec010caf4689b8c90449764f5cf9 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:43.242381 24514 leader_election.cc:302] T 442d4a29b1d743599f4b6b4871e26668 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:43.244297 24514 leader_election.cc:302] T 0b5df303cd6a4d59aca5c18bbbff24ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:43.244622 26362 raft_consensus.cc:2726] T 442d4a29b1d743599f4b6b4871e26668 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260925 03:04:43.245822 26362 raft_consensus.cc:2726] T 0b5df303cd6a4d59aca5c18bbbff24ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260925 03:04:43.246192 24514 leader_election.cc:302] T 7993f9c3e2a44f408bbf5d1e395e2306 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
W20260925 03:04:43.244861 26464 log.cc:924] Time spent T cc2df40db08649c9a75e90000b878559 P 869b0a8575b54d08ae2817fad5dd42d5: Append to log took a long time: real 0.333s user 0.001s sys 0.000s
I20260925 03:04:43.245141 26241 raft_consensus.cc:2781] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260925 03:04:43.219779 25942 raft_consensus.cc:3037] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:04:43.247114 24544 raft_consensus.cc:1261] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260925 03:04:43.201153 26450 raft_consensus.cc:504] T a4abe0ea757e4886be4945c5001e529a P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:43.227964 26247 raft_consensus.cc:3037] T a7aa47c48404465b94a36fd819fe3c7a P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:43.246044 26425 leader_election.cc:288] T 9e8af2641e124f9589c5ccfdac785f63 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:43.295097 24544 raft_consensus.cc:1226] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 4. Current term is 5. Ops: []
W20260925 03:04:43.298425 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 52303.5us
I20260925 03:04:43.260524 26241 raft_consensus.cc:482] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:43.305092 24514 leader_election.cc:398] T 420163a6cdce412ebc21c25fe3835638 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 election: Vote denied by peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187) with higher term. Message: Invalid argument: T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 4. Current term is 5.
I20260925 03:04:43.286867 26376 raft_consensus.cc:2726] T 7993f9c3e2a44f408bbf5d1e395e2306 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260925 03:04:43.260352 26348 consensus_queue.cc:1046] T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.041s
I20260925 03:04:43.320528 24544 raft_consensus.cc:1261] T b030d84c670b48d5bff337ccc0c77626 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260925 03:04:43.244889 26357 consensus_queue.cc:1046] T 43366a0a2f9d4344bf4e37ffd290ef8a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.053s
I20260925 03:04:43.303730 26241 raft_consensus.cc:3037] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:43.326103 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ac285175464fbda67211d9574db415" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
W20260925 03:04:43.330765 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 156145us
I20260925 03:04:43.219300 26448 raft_consensus.cc:504] T edcf6e6a422d4d198d4fbb0f7d3d845d P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:43.298024 26245 consensus_queue.cc:1035] T 37a157996e944db2906110d87ecb0c54 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.030s
I20260925 03:04:43.335835 24514 leader_election.cc:302] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
W20260925 03:04:43.326015 26472 log.cc:924] Time spent T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.219s user 0.000s sys 0.001s
W20260925 03:04:43.346043 26476 log.cc:924] Time spent T 100c113a9fd84cc8adf2ec7b33ea0093 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.088s user 0.001s sys 0.000s
I20260925 03:04:43.341846 26264 leader_election.cc:288] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:43.357924 26247 raft_consensus.cc:504] T a7aa47c48404465b94a36fd819fe3c7a P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:43.358213 26270 consensus_queue.cc:1035] T cc2df40db08649c9a75e90000b878559 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260925 03:04:43.359972 26247 leader_election.cc:288] T a7aa47c48404465b94a36fd819fe3c7a P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:43.361416 26406 leader_election.cc:288] T c85f5d1d3f004cf5817bb594fd8e6469 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:43.363492 26262 consensus_queue.cc:1046] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260925 03:04:43.364221 26450 leader_election.cc:288] T a4abe0ea757e4886be4945c5001e529a P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:43.364476 26264 raft_consensus.cc:3032] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c [term 3 LEADER]: Stepping down as leader of term 3
I20260925 03:04:43.364902 26264 raft_consensus.cc:729] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:43.365540 26373 leader_election.cc:288] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:43.365688 26264 consensus_queue.cc:260] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:04:43.367672 26474 log.cc:924] Time spent T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.147s user 0.001s sys 0.000s
I20260925 03:04:43.369292 24618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4abe0ea757e4886be4945c5001e529a" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:43.384555 24586 leader_election.cc:302] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 1bc1f2e62ab44af995165bd5116a9d43, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:43.385701 26401 raft_consensus.cc:482] T e92f377a79354a6ebb4eec67303c13e3 P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:43.390837 26401 raft_consensus.cc:504] T e92f377a79354a6ebb4eec67303c13e3 P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:43.393234 24586 leader_election.cc:302] T 7993f9c3e2a44f408bbf5d1e395e2306 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:04:43.394447 24586 leader_election.cc:302] T 550118bc1a0641ce9803bdbf43f2069f P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:04:43.395601 24586 leader_election.cc:302] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:04:43.402706 24586 leader_election.cc:302] T 21c43af5e6844fdaacc8c625fa01fd8d P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:43.403052 26441 raft_consensus.cc:2781] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260925 03:04:43.403119 26403 consensus_queue.cc:1035] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.010s
I20260925 03:04:43.404597 26441 raft_consensus.cc:482] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:43.404958 26441 raft_consensus.cc:3037] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:04:43.405791 26386 raft_consensus.cc:482] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:43.406533 26401 leader_election.cc:288] T e92f377a79354a6ebb4eec67303c13e3 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:43.406279 26386 raft_consensus.cc:504] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:43.407447 26421 raft_consensus.cc:686] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 LEADER]: Becoming Leader. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:04:43.408200 26386 leader_election.cc:288] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:43.408344 26421 consensus_queue.cc:237] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:43.409217 26483 raft_consensus.cc:2726] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260925 03:04:43.411517 26475 log.cc:924] Time spent T 21c43af5e6844fdaacc8c625fa01fd8d P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.122s user 0.000s sys 0.001s
I20260925 03:04:43.411643 26305 raft_consensus.cc:482] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:43.412195 26305 raft_consensus.cc:504] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:43.414656 26305 leader_election.cc:288] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
W20260925 03:04:43.415961 26293 consensus_meta.cc:287] T 0b5df303cd6a4d59aca5c18bbbff24ed P eeab93abfdb141a895002ec9f312296c: Time spent flushing consensus metadata: real 0.550s user 0.005s sys 0.002s
I20260925 03:04:43.418184 26439 raft_consensus.cc:504] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:43.419168 24618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60d45b7d467d4e1ab92367f6d7d52b15" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:43.418979 26379 raft_consensus.cc:504] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:43.420012 24470 raft_consensus.cc:1226] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 3. Current term is 4. Ops: []
I20260925 03:04:43.420046 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e8af2641e124f9589c5ccfdac785f63" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:43.396130 26348 raft_consensus.cc:2781] T 7993f9c3e2a44f408bbf5d1e395e2306 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260925 03:04:43.391558 26448 leader_election.cc:288] T edcf6e6a422d4d198d4fbb0f7d3d845d P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:43.424777 24615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e92f377a79354a6ebb4eec67303c13e3" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:43.424914 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7aa47c48404465b94a36fd819fe3c7a" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:43.360450 25979 raft_consensus.cc:504] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:43.403714 26247 raft_consensus.cc:2781] T 550118bc1a0641ce9803bdbf43f2069f P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader election won for term 4
I20260925 03:04:43.453080 24468 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0e54ca1757ee4e42956a9cebbeca4f62"
dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
mode: GRACEFUL
new_leader_uuid: "eeab93abfdb141a895002ec9f312296c"
from {username='slave'} at 127.0.0.1:38832
I20260925 03:04:43.453729 24468 raft_consensus.cc:595] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Received request to transfer leadership to TS eeab93abfdb141a895002ec9f312296c
I20260925 03:04:43.454087 24468 raft_consensus.cc:603] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260925 03:04:43.454746 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35992316cb8d4c028333388eff02ecde" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:43.455454 24472 raft_consensus.cc:2454] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 5.
I20260925 03:04:43.457927 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcf6e6a422d4d198d4fbb0f7d3d845d" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:43.458658 24466 raft_consensus.cc:2454] T edcf6e6a422d4d198d4fbb0f7d3d845d P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 2.
W20260925 03:04:43.461625 24436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 50054.4us
I20260925 03:04:43.424849 26348 raft_consensus.cc:482] T 7993f9c3e2a44f408bbf5d1e395e2306 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:43.466837 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6b6ec010caf4689b8c90449764f5cf9" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:43.450731 26381 raft_consensus.cc:2726] T c88cbd39e01f44628a094af516be2833 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260925 03:04:43.411273 26325 raft_consensus.cc:504] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:43.459304 26372 raft_consensus.cc:2726] T 21c43af5e6844fdaacc8c625fa01fd8d P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260925 03:04:43.465756 26409 raft_consensus.cc:686] T a7aa47c48404465b94a36fd819fe3c7a P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 LEADER]: Becoming Leader. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:04:43.473131 24473 raft_consensus.cc:1261] T 293ef69caf514ac98510a68495f54eda P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Refusing update from remote peer 1bc1f2e62ab44af995165bd5116a9d43: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260925 03:04:43.465306 26348 raft_consensus.cc:3037] T 7993f9c3e2a44f408bbf5d1e395e2306 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:43.463284 26439 leader_election.cc:288] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 7 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:43.487466 24464 raft_consensus.cc:1261] T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Refusing update from remote peer 1bc1f2e62ab44af995165bd5116a9d43: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260925 03:04:43.497294 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e54ca1757ee4e42956a9cebbeca4f62" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:43.497999 24468 raft_consensus.cc:3037] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:04:43.472404 26379 leader_election.cc:288] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:43.477200 26247 raft_consensus.cc:686] T 550118bc1a0641ce9803bdbf43f2069f P eeab93abfdb141a895002ec9f312296c [term 4 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:43.468930 26248 consensus_queue.cc:1035] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.029s
I20260925 03:04:43.510540 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367277daf8d04778b3fa4ed94d71580d" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:43.520673 24464 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1414dcfe53f64a78ae413531eb8ec117"
dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
mode: GRACEFUL
new_leader_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
from {username='slave'} at 127.0.0.1:38832
I20260925 03:04:43.521642 24464 raft_consensus.cc:595] T 1414dcfe53f64a78ae413531eb8ec117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Received request to transfer leadership to TS a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:43.522063 24464 raft_consensus.cc:603] T 1414dcfe53f64a78ae413531eb8ec117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260925 03:04:43.515141 26479 log.cc:924] Time spent T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5: Append to log took a long time: real 0.201s user 0.002s sys 0.000s
I20260925 03:04:43.502089 25191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367277daf8d04778b3fa4ed94d71580d" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
W20260925 03:04:43.524263 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 51949.3us
I20260925 03:04:43.517382 26441 raft_consensus.cc:504] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:43.526304 24619 raft_consensus.cc:1261] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260925 03:04:43.519304 26325 leader_election.cc:288] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
W20260925 03:04:43.524595 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 50221us
I20260925 03:04:43.517073 24615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e54ca1757ee4e42956a9cebbeca4f62" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:43.534730 24615 raft_consensus.cc:3037] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:04:43.480293 26246 consensus_queue.cc:1035] T 74443a0a1dc34d88a2f42e3b722db405 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.024s
W20260925 03:04:43.555511 25163 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 73830.8us
I20260925 03:04:43.555918 24618 raft_consensus.cc:2379] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 in current term 6: Already voted for candidate eeab93abfdb141a895002ec9f312296c in this term.
I20260925 03:04:43.556841 24588 leader_election.cc:302] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:43.561100 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f68804eb5a244d87bb395d50683c6012" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:43.498355 26439 consensus_queue.cc:1035] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.036s
I20260925 03:04:43.563278 24620 raft_consensus.cc:1261] T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260925 03:04:43.565454 24588 leader_election.cc:302] T edcf6e6a422d4d198d4fbb0f7d3d845d P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:43.558161 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:43.566491 24619 raft_consensus.cc:3037] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:04:43.562578 26409 consensus_queue.cc:237] T a7aa47c48404465b94a36fd819fe3c7a P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:04:43.570223 25165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 84578.3us
I20260925 03:04:43.575605 24436 leader_election.cc:302] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: 1bc1f2e62ab44af995165bd5116a9d43, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:43.579766 24464 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "48ca6ccd98194cf2af7b5278d15bf4ea"
dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
mode: GRACEFUL
new_leader_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
from {username='slave'} at 127.0.0.1:38832
I20260925 03:04:43.580444 24464 raft_consensus.cc:595] T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Received request to transfer leadership to TS a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:43.580825 24464 raft_consensus.cc:603] T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260925 03:04:43.519622 26247 consensus_queue.cc:237] T 550118bc1a0641ce9803bdbf43f2069f P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:43.538146 26293 consensus_queue.cc:1046] T 48ca6ccd98194cf2af7b5278d15bf4ea P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.031s
I20260925 03:04:43.573784 24615 raft_consensus.cc:2454] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 7.
I20260925 03:04:43.562628 25979 leader_election.cc:288] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 7 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:43.567770 26483 consensus_queue.cc:1035] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260925 03:04:43.593201 25198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43"
I20260925 03:04:43.569553 26367 raft_consensus.cc:2781] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260925 03:04:43.597402 25198 raft_consensus.cc:3037] T 29637f5a4a504ce6ba29033ccd0cd716 P 1bc1f2e62ab44af995165bd5116a9d43 [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:04:43.596567 24468 raft_consensus.cc:2454] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 7.
I20260925 03:04:43.601115 25162 leader_election.cc:302] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:43.564679 26262 consensus_queue.cc:1035] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.030s
I20260925 03:04:43.585232 26380 raft_consensus.cc:482] T 43366a0a2f9d4344bf4e37ffd290ef8a P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:43.605605 26380 raft_consensus.cc:504] T 43366a0a2f9d4344bf4e37ffd290ef8a P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:43.607772 26380 leader_election.cc:288] T 43366a0a2f9d4344bf4e37ffd290ef8a P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:43.612850 25198 raft_consensus.cc:2454] T 29637f5a4a504ce6ba29033ccd0cd716 P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 6.
I20260925 03:04:43.583461 26426 raft_consensus.cc:2726] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260925 03:04:43.584224 26386 raft_consensus.cc:2781] T edcf6e6a422d4d198d4fbb0f7d3d845d P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260925 03:04:43.572764 26325 consensus_queue.cc:1035] T 0be4e0aa984f4f2b81a58e382e434ad5 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.027s
I20260925 03:04:43.570346 26379 consensus_queue.cc:1035] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.028s
W20260925 03:04:43.623071 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 62703us
I20260925 03:04:43.564673 26480 consensus_queue.cc:1046] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.034s
I20260925 03:04:43.624966 24468 raft_consensus.cc:1226] T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 2. Current term is 3. Ops: []
I20260925 03:04:43.618475 25191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5aec38e66a947c681e6c2a72b05e6ca" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43"
I20260925 03:04:43.626518 24465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5aec38e66a947c681e6c2a72b05e6ca" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:43.599259 26367 raft_consensus.cc:482] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Starting leader election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
W20260925 03:04:43.626565 26493 log.cc:924] Time spent T c88cbd39e01f44628a094af516be2833 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.067s user 0.001s sys 0.000s
I20260925 03:04:43.627676 24464 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "550118bc1a0641ce9803bdbf43f2069f"
dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
mode: GRACEFUL
new_leader_uuid: "eeab93abfdb141a895002ec9f312296c"
from {username='slave'} at 127.0.0.1:38832
I20260925 03:04:43.579001 26413 raft_consensus.cc:504] T b1e85248d3b04b979dd080caeeb02fdb P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:43.634156 25191 raft_consensus.cc:3037] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:04:43.634263 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60d45b7d467d4e1ab92367f6d7d52b15" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:43.634850 24469 raft_consensus.cc:3037] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:04:43.635594 26413 leader_election.cc:288] T b1e85248d3b04b979dd080caeeb02fdb P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:43.629346 26441 leader_election.cc:288] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 6 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
W20260925 03:04:43.629963 26491 log.cc:924] Time spent T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.133s user 0.000s sys 0.001s
I20260925 03:04:43.630713 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43366a0a2f9d4344bf4e37ffd290ef8a" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:43.631346 24465 raft_consensus.cc:3037] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Advancing to term 7
W20260925 03:04:43.627143 26488 log.cc:924] Time spent T ac9357d144aa49789faaf31fe5e6bf89 P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.126s user 0.000s sys 0.000s
I20260925 03:04:43.621134 26386 raft_consensus.cc:482] T edcf6e6a422d4d198d4fbb0f7d3d845d P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:43.632580 24464 raft_consensus.cc:595] T 550118bc1a0641ce9803bdbf43f2069f P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Received request to transfer leadership to TS eeab93abfdb141a895002ec9f312296c
I20260925 03:04:43.642091 24464 raft_consensus.cc:603] T 550118bc1a0641ce9803bdbf43f2069f P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260925 03:04:43.642972 24473 raft_consensus.cc:2379] T 43366a0a2f9d4344bf4e37ffd290ef8a P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c in current term 3: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:04:43.585572 26450 consensus_queue.cc:1035] T b1e85248d3b04b979dd080caeeb02fdb P 1bc1f2e62ab44af995165bd5116a9d43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.035s
I20260925 03:04:43.643317 26386 raft_consensus.cc:3037] T edcf6e6a422d4d198d4fbb0f7d3d845d P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:43.650830 25196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1e85248d3b04b979dd080caeeb02fdb" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43"
I20260925 03:04:43.658090 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b5df303cd6a4d59aca5c18bbbff24ed" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:43.633098 26367 raft_consensus.cc:3037] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:04:43.671373 26264 raft_consensus.cc:3037] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:43.661304 24465 raft_consensus.cc:2454] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 7.
I20260925 03:04:43.661731 24614 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e92f377a79354a6ebb4eec67303c13e3"
dest_uuid: "eeab93abfdb141a895002ec9f312296c"
mode: GRACEFUL
new_leader_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
from {username='slave'} at 127.0.0.1:44570
I20260925 03:04:43.675494 24614 raft_consensus.cc:595] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [term 4 LEADER]: Received request to transfer leadership to TS a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:43.656248 26348 raft_consensus.cc:504] T 7993f9c3e2a44f408bbf5d1e395e2306 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:43.662680 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06f4a7eaf6534e4986c9fd35911038e4" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:43.663441 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:43.682281 24542 raft_consensus.cc:2454] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 5.
I20260925 03:04:43.664155 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e5819c15eac4544a57e5ee6c9d4f117" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:43.669344 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48ca6ccd98194cf2af7b5278d15bf4ea" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:43.670413 24546 raft_consensus.cc:1261] T 21c43af5e6844fdaacc8c625fa01fd8d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20260925 03:04:43.684079 24546 pending_rounds.cc:76] T 21c43af5e6844fdaacc8c625fa01fd8d P a04a37fc4aa7460cbfdbeeb4e451e1f5: Aborting all ops after (but not including) 1
I20260925 03:04:43.684329 24546 pending_rounds.cc:97] T 21c43af5e6844fdaacc8c625fa01fd8d P a04a37fc4aa7460cbfdbeeb4e451e1f5: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260925 03:04:43.684547 24546 raft_consensus.cc:2866] T 21c43af5e6844fdaacc8c625fa01fd8d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260925 03:04:43.632624 26337 raft_consensus.cc:2781] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [term 7 FOLLOWER]: Leader election won for term 7
I20260925 03:04:43.680543 24619 raft_consensus.cc:2454] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 6.
I20260925 03:04:43.687438 24544 raft_consensus.cc:2379] T 48ca6ccd98194cf2af7b5278d15bf4ea P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 in current term 3: Already voted for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in this term.
I20260925 03:04:43.689886 24469 raft_consensus.cc:2396] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260925 03:04:43.703126 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60d45b7d467d4e1ab92367f6d7d52b15" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:43.703964 24542 raft_consensus.cc:2454] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 5.
W20260925 03:04:43.713255 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 73870.9us
W20260925 03:04:43.713440 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 81511.8us
I20260925 03:04:43.715173 24436 leader_election.cc:302] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:43.718035 24546 raft_consensus.cc:1261] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260925 03:04:43.719322 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37a157996e944db2906110d87ecb0c54" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:43.720125 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e8af2641e124f9589c5ccfdac785f63" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:43.720669 24547 raft_consensus.cc:2379] T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 in current term 4: Already voted for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in this term.
I20260925 03:04:43.720999 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7691c6b475428c82d6181d693d26f3" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:43.724742 24545 raft_consensus.cc:1261] T 0be4e0aa984f4f2b81a58e382e434ad5 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260925 03:04:43.713203 26337 raft_consensus.cc:686] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [term 7 LEADER]: Becoming Leader. State: Replica: 1bc1f2e62ab44af995165bd5116a9d43, State: Running, Role: LEADER
I20260925 03:04:43.675812 26241 raft_consensus.cc:504] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:43.726480 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367277daf8d04778b3fa4ed94d71580d" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:43.727239 24546 raft_consensus.cc:2454] T 367277daf8d04778b3fa4ed94d71580d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 4.
I20260925 03:04:43.731273 25191 raft_consensus.cc:2454] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 7.
I20260925 03:04:43.719530 26498 raft_consensus.cc:2781] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader election won for term 6
I20260925 03:04:43.733090 26498 raft_consensus.cc:686] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 LEADER]: Becoming Leader. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:04:43.734067 26498 consensus_queue.cc:237] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 6, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:43.739364 26241 leader_election.cc:288] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:43.748265 24619 raft_consensus.cc:1226] T 7993f9c3e2a44f408bbf5d1e395e2306 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 4. Current term is 5. Ops: []
I20260925 03:04:43.749399 25191 raft_consensus.cc:1261] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260925 03:04:43.750560 24541 raft_consensus.cc:1226] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 4. Current term is 5. Ops: []
I20260925 03:04:43.756788 24437 leader_election.cc:302] T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:43.751166 23300 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260925 03:04:43.758183 24437 leader_election.cc:302] T 8e5819c15eac4544a57e5ee6c9d4f117 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:43.748428 24585 leader_election.cc:302] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:43.759383 24437 leader_election.cc:302] T 06f4a7eaf6534e4986c9fd35911038e4 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:43.753412 26348 leader_election.cc:288] T 7993f9c3e2a44f408bbf5d1e395e2306 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:43.754467 26366 raft_consensus.cc:3032] T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c [term 4 LEADER]: Stepping down as leader of term 4
I20260925 03:04:43.755182 26372 raft_consensus.cc:3032] T 48ca6ccd98194cf2af7b5278d15bf4ea P eeab93abfdb141a895002ec9f312296c [term 2 LEADER]: Stepping down as leader of term 2
I20260925 03:04:43.756034 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f68804eb5a244d87bb395d50683c6012" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:43.763079 24437 leader_election.cc:302] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:04:43.753407 25196 raft_consensus.cc:1261] T 71f461a6e2ff434993b78a2d077f0abd P 1bc1f2e62ab44af995165bd5116a9d43 [term 5 FOLLOWER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260925 03:04:43.767931 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:43.768601 24544 raft_consensus.cc:3037] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:04:43.765503 26366 raft_consensus.cc:729] T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c [term 4 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:43.771428 24437 leader_election.cc:302] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:43.769881 26421 raft_consensus.cc:2741] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20260925 03:04:43.764459 26422 raft_consensus.cc:2726] T 06f4a7eaf6534e4986c9fd35911038e4 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260925 03:04:43.765341 26372 raft_consensus.cc:729] T 48ca6ccd98194cf2af7b5278d15bf4ea P eeab93abfdb141a895002ec9f312296c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:43.777423 26366 consensus_queue.cc:260] T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:43.778855 26366 raft_consensus.cc:3037] T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:43.778424 26372 consensus_queue.cc:260] T 48ca6ccd98194cf2af7b5278d15bf4ea P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:43.764469 26365 raft_consensus.cc:2726] T 8e5819c15eac4544a57e5ee6c9d4f117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260925 03:04:43.770704 24545 raft_consensus.cc:1261] T 5278a60f97da4e10aecd50a8d9a9a43c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260925 03:04:43.762395 26497 raft_consensus.cc:2726] T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260925 03:04:43.785769 24437 leader_election.cc:302] T 37a157996e944db2906110d87ecb0c54 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
W20260925 03:04:43.781391 26510 log.cc:924] Time spent T 21c43af5e6844fdaacc8c625fa01fd8d P 869b0a8575b54d08ae2817fad5dd42d5: Append to log took a long time: real 0.057s user 0.000s sys 0.001s
I20260925 03:04:43.791745 24544 raft_consensus.cc:2454] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 6.
I20260925 03:04:43.794100 24437 leader_election.cc:302] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:43.795064 26508 consensus_queue.cc:1046] T 48ca6ccd98194cf2af7b5278d15bf4ea P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260925 03:04:43.796142 26508 consensus_queue.cc:1046] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260925 03:04:43.797108 26508 raft_consensus.cc:3032] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c [term 3 LEADER]: Stepping down as leader of term 3
I20260925 03:04:43.797402 26508 raft_consensus.cc:729] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:43.798777 24619 raft_consensus.cc:1261] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260925 03:04:43.799062 24585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.4:43979, user_credentials={real_user=slave}} blocked reactor thread for 50714.6us
I20260925 03:04:43.799299 26262 raft_consensus.cc:2781] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Leader election won for term 7
I20260925 03:04:43.799628 26374 raft_consensus.cc:482] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:43.805100 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceab6b0fccd54e909a64f66fe81a9a1f" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:43.800107 26374 raft_consensus.cc:504] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:43.781627 26305 consensus_queue.cc:1035] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.025s
I20260925 03:04:43.806092 26421 consensus_queue.cc:1046] T 7993f9c3e2a44f408bbf5d1e395e2306 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260925 03:04:43.782917 26409 consensus_queue.cc:1035] T 21c43af5e6844fdaacc8c625fa01fd8d P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.021s
I20260925 03:04:43.809098 24465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7993f9c3e2a44f408bbf5d1e395e2306" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:43.810292 24616 raft_consensus.cc:1261] T 5278a60f97da4e10aecd50a8d9a9a43c P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260925 03:04:43.785262 26337 consensus_queue.cc:237] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:43.823000 24617 raft_consensus.cc:2454] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 4.
I20260925 03:04:43.812307 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f1fac028a144058b92331c0d9b44ebf" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:43.807195 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1e85248d3b04b979dd080caeeb02fdb" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:43.820154 26325 consensus_queue.cc:1035] T 0be4e0aa984f4f2b81a58e382e434ad5 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260925 03:04:43.794184 26426 raft_consensus.cc:2726] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260925 03:04:43.811434 26422 raft_consensus.cc:2726] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260925 03:04:43.835115 26371 raft_consensus.cc:482] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:43.803882 26365 raft_consensus.cc:2726] T 37a157996e944db2906110d87ecb0c54 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260925 03:04:43.846899 24616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c85f5d1d3f004cf5817bb594fd8e6469" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:43.847744 26371 raft_consensus.cc:504] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:43.849925 26371 leader_election.cc:288] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:43.832964 26421 consensus_queue.cc:1225] T 7993f9c3e2a44f408bbf5d1e395e2306 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Peer eeab93abfdb141a895002ec9f312296c log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260925 03:04:43.852939 24614 raft_consensus.cc:1261] T 71f461a6e2ff434993b78a2d077f0abd P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260925 03:04:43.854444 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7691c6b475428c82d6181d693d26f3" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:43.855199 24622 raft_consensus.cc:2454] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 6.
I20260925 03:04:43.860188 24468 raft_consensus.cc:1261] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 3. (term mismatch)
I20260925 03:04:43.860646 24468 pending_rounds.cc:76] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5: Aborting all ops after (but not including) 2
I20260925 03:04:43.860874 24468 pending_rounds.cc:97] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5: Aborting uncommitted NO_OP operation due to leader change: 5.3
I20260925 03:04:43.861115 24468 raft_consensus.cc:2866] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260925 03:04:43.862448 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b030d84c670b48d5bff337ccc0c77626" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:43.798152 26508 consensus_queue.cc:260] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:43.864739 26372 raft_consensus.cc:3037] T 48ca6ccd98194cf2af7b5278d15bf4ea P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:43.832558 26262 raft_consensus.cc:686] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [term 7 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:43.866169 26262 consensus_queue.cc:237] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:43.867379 24588 leader_election.cc:302] T b1e85248d3b04b979dd080caeeb02fdb P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:04:43.868382 26448 raft_consensus.cc:2726] T b1e85248d3b04b979dd080caeeb02fdb P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260925 03:04:43.874456 26374 leader_election.cc:288] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:43.877731 24620 raft_consensus.cc:1226] T b1e85248d3b04b979dd080caeeb02fdb P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Rejecting Update request from peer 1bc1f2e62ab44af995165bd5116a9d43 for earlier term 4. Current term is 5. Ops: []
I20260925 03:04:43.857487 24618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06f4a7eaf6534e4986c9fd35911038e4" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:43.884411 24464 raft_consensus.cc:1261] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260925 03:04:43.852555 26325 consensus_queue.cc:1035] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.035s
I20260925 03:04:43.894714 26386 raft_consensus.cc:504] T edcf6e6a422d4d198d4fbb0f7d3d845d P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:43.897770 26386 leader_election.cc:288] T edcf6e6a422d4d198d4fbb0f7d3d845d P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 3 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:43.904234 26427 raft_consensus.cc:3037] T 25102ba08e4e49388f6176badcd10a25 P eeab93abfdb141a895002ec9f312296c [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:04:43.906996 24620 raft_consensus.cc:1261] T 420163a6cdce412ebc21c25fe3835638 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260925 03:04:43.910311 24548 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 811) took 1064 ms. Trace:
I20260925 03:04:43.914155 24436 leader_election.cc:302] T 9e8af2641e124f9589c5ccfdac785f63 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:43.910727 24548 rpcz_store.cc:276] 0925 03:04:42.845904 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:42.948227 (+102323us) service_pool.cc:227] Handling call
0925 03:04:42.948620 (+ 393us) raft_consensus.cc:1457] Updating replica for 2 ops
0925 03:04:42.963629 (+ 15009us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:42.963662 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0925 03:04:42.970386 (+ 6724us) log.cc:842] Serialized 52 byte log entry
0925 03:04:43.198528 (+228142us) raft_consensus.cc:1631] Marking committed up to 3
0925 03:04:43.313302 (+114774us) log.cc:842] Serialized 14 byte log entry
0925 03:04:43.320225 (+ 6923us) log.cc:842] Serialized 14 byte log entry
0925 03:04:43.909743 (+589518us) spinlock_profiling.cc:243] Waited 1.68 ms on lock 0x7b18002e1260. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f437c4478b1 00007f437c454661 00007f437351f4ca 00007f437cc00f99 00007f43733e3990 00007f43735a5864 00007f43735a5155 00007f43764c83d0 00007f43764da576 00007f43764eb0d7 00007f43764eb058
0925 03:04:43.909910 (+ 167us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:43.909978 (+ 68us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:04:43.910121 (+ 143us) raft_consensus.cc:1671] finished
0925 03:04:43.910136 (+ 15us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:43.910256 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8007,"prepare.queue_time_us":191423,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":49,"raft.queue_time_us":14429,"raft.run_cpu_time_us":270,"raft.run_wall_time_us":15759,"spinlock_wait_cycles":3867392,"thread_start_us":248229,"threads_started":2,"wal-append.queue_time_us":205813}
I20260925 03:04:43.917740 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5f9b503b0ab43999d582015a367441e" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:43.920109 26362 raft_consensus.cc:3032] T 43366a0a2f9d4344bf4e37ffd290ef8a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 LEADER]: Stepping down as leader of term 2
W20260925 03:04:43.921139 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 588423us
I20260925 03:04:43.921379 26499 raft_consensus.cc:2781] T 9e8af2641e124f9589c5ccfdac785f63 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260925 03:04:43.921784 26499 raft_consensus.cc:482] T 9e8af2641e124f9589c5ccfdac785f63 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:43.922049 26376 raft_consensus.cc:2726] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260925 03:04:43.920498 26362 raft_consensus.cc:729] T 43366a0a2f9d4344bf4e37ffd290ef8a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:04:43.915701 26264 raft_consensus.cc:482] T 0b5df303cd6a4d59aca5c18bbbff24ed P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:43.923264 26264 raft_consensus.cc:504] T 0b5df303cd6a4d59aca5c18bbbff24ed P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:43.923450 26362 consensus_queue.cc:260] T 43366a0a2f9d4344bf4e37ffd290ef8a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:43.924629 26362 raft_consensus.cc:3037] T 43366a0a2f9d4344bf4e37ffd290ef8a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:43.925595 26264 leader_election.cc:288] T 0b5df303cd6a4d59aca5c18bbbff24ed P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:43.922120 26499 raft_consensus.cc:3037] T 9e8af2641e124f9589c5ccfdac785f63 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:43.893167 26367 raft_consensus.cc:504] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:43.918530 24618 raft_consensus.cc:2454] T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 5.
I20260925 03:04:43.933329 24514 leader_election.cc:302] T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
W20260925 03:04:43.915673 26377 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.971s user 0.026s sys 0.038s
I20260925 03:04:43.935563 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcf6e6a422d4d198d4fbb0f7d3d845d" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:43.949966 24466 raft_consensus.cc:3037] T edcf6e6a422d4d198d4fbb0f7d3d845d P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:43.954239 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcf6e6a422d4d198d4fbb0f7d3d845d" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:43.954990 24546 raft_consensus.cc:2454] T edcf6e6a422d4d198d4fbb0f7d3d845d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 2.
I20260925 03:04:43.942781 26363 raft_consensus.cc:482] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260925 03:04:43.944197 25162 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 55839.4us
I20260925 03:04:43.945007 26372 raft_consensus.cc:482] T 420163a6cdce412ebc21c25fe3835638 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:43.946906 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ac285175464fbda67211d9574db415" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:43.955611 24437 leader_election.cc:302] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:43.964042 24466 raft_consensus.cc:2454] T edcf6e6a422d4d198d4fbb0f7d3d845d P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 3.
I20260925 03:04:43.956295 26412 raft_consensus.cc:482] T 0be4e0aa984f4f2b81a58e382e434ad5 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:43.955717 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7aa47c48404465b94a36fd819fe3c7a" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:43.957664 24543 raft_consensus.cc:1226] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 4. Current term is 5. Ops: []
I20260925 03:04:43.942260 26357 raft_consensus.cc:482] T b030d84c670b48d5bff337ccc0c77626 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:43.955403 26368 raft_consensus.cc:482] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:43.956712 26241 raft_consensus.cc:482] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:44.008366 24545 raft_consensus.cc:3037] T a7aa47c48404465b94a36fd819fe3c7a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:43.977017 24621 raft_consensus.cc:1226] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 5. Current term is 6. Ops: []
I20260925 03:04:43.978304 26426 raft_consensus.cc:2726] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260925 03:04:43.978819 26438 consensus_queue.cc:1035] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260925 03:04:43.979044 26363 raft_consensus.cc:504] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:43.957664 26406 consensus_queue.cc:1035] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.022s
I20260925 03:04:43.982007 26386 consensus_queue.cc:1035] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260925 03:04:43.983930 26383 raft_consensus.cc:3037] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:43.985759 24511 leader_election.cc:302] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
W20260925 03:04:44.057847 24511 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 72141.2us
I20260925 03:04:43.986356 26376 consensus_queue.cc:1035] T 8ac466793d094325ace224708b2158ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20260925 03:04:43.989923 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 62446.9us
I20260925 03:04:44.006575 24541 raft_consensus.cc:2454] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 4.
I20260925 03:04:43.994570 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b5df303cd6a4d59aca5c18bbbff24ed" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:44.065251 24466 raft_consensus.cc:2454] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 5.
I20260925 03:04:44.008755 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43366a0a2f9d4344bf4e37ffd290ef8a" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:43.961998 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35992316cb8d4c028333388eff02ecde" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
W20260925 03:04:44.010437 25165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 245026us
W20260925 03:04:43.949734 26377 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.005s user 0.027s sys 0.038s
I20260925 03:04:44.013339 26357 raft_consensus.cc:504] T b030d84c670b48d5bff337ccc0c77626 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:44.054666 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60d45b7d467d4e1ab92367f6d7d52b15" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:44.061380 24514 leader_election.cc:302] T 25102ba08e4e49388f6176badcd10a25 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:43.978667 26381 raft_consensus.cc:2726] T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260925 03:04:44.069960 24543 raft_consensus.cc:3037] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:04:43.949121 26371 consensus_queue.cc:1035] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.097s
W20260925 03:04:44.063570 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 49736.4us
I20260925 03:04:44.016532 26432 consensus_queue.cc:1035] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.055s
I20260925 03:04:44.074566 26432 consensus_queue.cc:1225] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer 869b0a8575b54d08ae2817fad5dd42d5 log is divergent from this leader: its last log entry 3.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
W20260925 03:04:44.076025 26521 log.cc:924] Time spent T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43: Append to log took a long time: real 0.111s user 0.000s sys 0.000s
I20260925 03:04:44.076870 25196 raft_consensus.cc:1261] T e6b6ec010caf4689b8c90449764f5cf9 P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260925 03:04:44.078437 26367 leader_election.cc:288] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 6 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:44.013191 26412 raft_consensus.cc:504] T 0be4e0aa984f4f2b81a58e382e434ad5 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:44.078750 26381 consensus_queue.cc:1035] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260925 03:04:44.080238 26357 leader_election.cc:288] T b030d84c670b48d5bff337ccc0c77626 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:44.080546 26412 leader_election.cc:288] T 0be4e0aa984f4f2b81a58e382e434ad5 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:44.081048 26357 raft_consensus.cc:2726] T 25102ba08e4e49388f6176badcd10a25 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260925 03:04:44.082217 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7993f9c3e2a44f408bbf5d1e395e2306" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:44.082806 26379 consensus_queue.cc:1035] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260925 03:04:44.085289 25191 raft_consensus.cc:1261] T e92f377a79354a6ebb4eec67303c13e3 P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260925 03:04:44.082826 24548 raft_consensus.cc:3037] T 7993f9c3e2a44f408bbf5d1e395e2306 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:44.091580 26377 consensus_queue.cc:1035] T e6b6ec010caf4689b8c90449764f5cf9 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260925 03:04:44.099740 24588 leader_election.cc:302] T edcf6e6a422d4d198d4fbb0f7d3d845d P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:44.008083 26372 raft_consensus.cc:504] T 420163a6cdce412ebc21c25fe3835638 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:44.009197 26368 raft_consensus.cc:504] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:44.058199 26421 raft_consensus.cc:482] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:43.993222 26518 consensus_queue.cc:1046] T b1e85248d3b04b979dd080caeeb02fdb P 1bc1f2e62ab44af995165bd5116a9d43 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.096s
I20260925 03:04:44.100593 24588 leader_election.cc:302] T 0b5df303cd6a4d59aca5c18bbbff24ed P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:44.064292 26373 raft_consensus.cc:2781] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260925 03:04:44.112800 24548 raft_consensus.cc:2454] T 7993f9c3e2a44f408bbf5d1e395e2306 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 5.
I20260925 03:04:44.113586 26507 raft_consensus.cc:2781] T edcf6e6a422d4d198d4fbb0f7d3d845d P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader election won for term 3
I20260925 03:04:44.120081 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b5df303cd6a4d59aca5c18bbbff24ed" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:44.125142 24545 raft_consensus.cc:2454] T a7aa47c48404465b94a36fd819fe3c7a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 4.
I20260925 03:04:44.126420 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcf6e6a422d4d198d4fbb0f7d3d845d" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:44.127063 24545 raft_consensus.cc:3037] T edcf6e6a422d4d198d4fbb0f7d3d845d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:44.014456 26241 raft_consensus.cc:504] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
W20260925 03:04:44.116953 26496 log.cc:924] Time spent T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5: Append to log took a long time: real 0.576s user 0.006s sys 0.000s
I20260925 03:04:44.143210 24543 raft_consensus.cc:2454] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 6.
I20260925 03:04:44.144598 24586 leader_election.cc:302] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5
W20260925 03:04:44.150274 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 88981.7us
I20260925 03:04:44.130558 26518 consensus_queue.cc:1225] T b1e85248d3b04b979dd080caeeb02fdb P 1bc1f2e62ab44af995165bd5116a9d43 [LEADER]: Peer eeab93abfdb141a895002ec9f312296c log is divergent from this leader: its last log entry 3.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260925 03:04:44.128896 26373 raft_consensus.cc:482] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:44.154099 24546 raft_consensus.cc:2454] T 43366a0a2f9d4344bf4e37ffd290ef8a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 3.
I20260925 03:04:44.157492 26368 leader_election.cc:288] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:44.157917 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35992316cb8d4c028333388eff02ecde" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:44.142699 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35992316cb8d4c028333388eff02ecde" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:44.161290 26386 raft_consensus.cc:2781] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260925 03:04:44.164535 26386 raft_consensus.cc:482] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting leader election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:44.164938 26386 raft_consensus.cc:3037] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:44.168254 24468 pending_rounds.cc:76] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5: Aborting all ops after (but not including) 2
I20260925 03:04:44.168583 24468 pending_rounds.cc:97] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20260925 03:04:44.164016 24473 raft_consensus.cc:3037] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:04:44.161551 24586 leader_election.cc:302] T 7993f9c3e2a44f408bbf5d1e395e2306 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:04:44.162575 26270 raft_consensus.cc:482] T 25102ba08e4e49388f6176badcd10a25 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:44.168857 24468 raft_consensus.cc:2866] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260925 03:04:44.174971 24586 leader_election.cc:302] T a7aa47c48404465b94a36fd819fe3c7a P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:44.185933 24586 leader_election.cc:302] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:04:44.178771 24545 raft_consensus.cc:2454] T edcf6e6a422d4d198d4fbb0f7d3d845d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 3.
I20260925 03:04:44.163440 26373 raft_consensus.cc:3037] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Advancing to term 6
W20260925 03:04:44.199077 24511 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 140448us
I20260925 03:04:44.163247 26363 leader_election.cc:288] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:44.203081 24511 leader_election.cc:302] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:44.163211 26241 leader_election.cc:288] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:44.133430 26441 consensus_queue.cc:1035] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.054s
I20260925 03:04:44.151592 26507 raft_consensus.cc:686] T edcf6e6a422d4d198d4fbb0f7d3d845d P eeab93abfdb141a895002ec9f312296c [term 3 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:44.223122 24467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b8e0d260a884ecdb20f79de46c36b19" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:44.228536 24473 raft_consensus.cc:2454] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 6.
I20260925 03:04:44.230767 24588 leader_election.cc:302] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:44.200443 26413 raft_consensus.cc:2781] T a7aa47c48404465b94a36fd819fe3c7a P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader election won for term 4
I20260925 03:04:44.193136 26427 raft_consensus.cc:2781] T 7993f9c3e2a44f408bbf5d1e395e2306 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Leader election won for term 5
I20260925 03:04:44.191758 26414 raft_consensus.cc:482] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:44.258060 24540 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 821) took 1055 ms. Trace:
I20260925 03:04:44.204708 26262 raft_consensus.cc:2781] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Leader election won for term 6
I20260925 03:04:44.262683 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4ebd2b043024a548005c8b6b00b0449" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:44.129258 26421 raft_consensus.cc:504] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:44.265437 24465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e8af2641e124f9589c5ccfdac785f63" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:44.258548 24540 rpcz_store.cc:276] 0925 03:04:43.181851 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:43.184735 (+ 2884us) service_pool.cc:227] Handling call
0925 03:04:43.185018 (+ 283us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:43.228898 (+ 43880us) raft_consensus.cc:1520] Early marking committed up to index 3
0925 03:04:43.333760 (+104862us) log.cc:842] Serialized 14 byte log entry
0925 03:04:43.931642 (+597882us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:43.931694 (+ 52us) raft_consensus.cc:1631] Marking committed up to 3
0925 03:04:43.931735 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:44.236790 (+305055us) spinlock_profiling.cc:243] Waited 43.6 ms on lock 0x7b54003b8154. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d0f51 00007f43764cf99e 00007f437b4112e9 00007f4374af7684 00007f4374af7612 00007f4374af7541 00007f4374af74bc 00007f4374af6772 00007f43748fa94c 00007f43748f9d86
0925 03:04:44.236833 (+ 43us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:44.236961 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":473569,"prepare.queue_time_us":138095,"prepare.run_cpu_time_us":24,"prepare.run_wall_time_us":25,"raft.queue_time_us":484004,"raft.run_cpu_time_us":89,"raft.run_wall_time_us":90,"spinlock_wait_cycles":100335872,"thread_start_us":198172,"threads_started":2,"wal-append.queue_time_us":167050}
I20260925 03:04:44.267663 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b627ecb4a5384ef392aa695e06599c39" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:44.260881 25979 raft_consensus.cc:2781] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Leader election won for term 6
I20260925 03:04:44.277282 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48ca6ccd98194cf2af7b5278d15bf4ea" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:44.278131 24472 raft_consensus.cc:2379] T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 3: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:04:44.279206 24467 raft_consensus.cc:1226] T 21c43af5e6844fdaacc8c625fa01fd8d P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 LEADER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 2. Current term is 3. Ops: []
I20260925 03:04:44.280272 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b030d84c670b48d5bff337ccc0c77626" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:44.282434 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c85f5d1d3f004cf5817bb594fd8e6469" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:44.283593 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "293ef69caf514ac98510a68495f54eda" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:44.284680 24465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "550118bc1a0641ce9803bdbf43f2069f" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:44.285403 24465 raft_consensus.cc:2454] T 550118bc1a0641ce9803bdbf43f2069f P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 4.
I20260925 03:04:44.286254 24465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7691c6b475428c82d6181d693d26f3" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:44.286953 24465 raft_consensus.cc:2379] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 6: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:04:44.193243 26270 raft_consensus.cc:504] T 25102ba08e4e49388f6176badcd10a25 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:44.290625 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "420163a6cdce412ebc21c25fe3835638" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:44.291550 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5f9b503b0ab43999d582015a367441e" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:44.293320 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c23337a7cb74184a0e57b7095cb6a11" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:44.264082 26376 raft_consensus.cc:2781] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader election won for term 5
I20260925 03:04:44.173957 26448 consensus_queue.cc:1046] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.063s
I20260925 03:04:44.294420 24465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceab6b0fccd54e909a64f66fe81a9a1f" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:44.299181 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06f4a7eaf6534e4986c9fd35911038e4" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:44.301077 24616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be4e0aa984f4f2b81a58e382e434ad5" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:44.270740 26293 raft_consensus.cc:3032] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [term 4 LEADER]: Stepping down as leader of term 4
I20260925 03:04:44.304311 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e725b82985746b39c0cc8f671c95353" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:44.310235 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5f9b503b0ab43999d582015a367441e" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:44.299924 24437 leader_election.cc:302] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:44.305550 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:44.307374 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f1fac028a144058b92331c0d9b44ebf" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
W20260925 03:04:44.394796 25163 leader_election.cc:334] T 367277daf8d04778b3fa4ed94d71580d P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389): Timed out: RequestConsensusVote RPC to 127.22.193.2:36389 timed out after 1.645s (SENT)
I20260925 03:04:44.308833 24464 raft_consensus.cc:2379] T 06f4a7eaf6534e4986c9fd35911038e4 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 5: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:04:44.395135 24467 raft_consensus.cc:1226] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 5. Current term is 6. Ops: []
I20260925 03:04:44.215257 26400 consensus_queue.cc:1046] T 04e068a0f80941dc9138cee063f53bab P 1bc1f2e62ab44af995165bd5116a9d43 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.061s
I20260925 03:04:44.395898 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b030d84c670b48d5bff337ccc0c77626" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:44.311220 24622 raft_consensus.cc:2454] T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 5.
W20260925 03:04:44.396862 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 97013.7us
I20260925 03:04:44.397302 26262 raft_consensus.cc:686] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c [term 6 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
W20260925 03:04:44.398751 26537 log.cc:924] Time spent T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.086s user 0.000s sys 0.000s
I20260925 03:04:44.399287 26413 raft_consensus.cc:686] T a7aa47c48404465b94a36fd819fe3c7a P eeab93abfdb141a895002ec9f312296c [term 4 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:44.400549 26451 raft_consensus.cc:2781] T 0b5df303cd6a4d59aca5c18bbbff24ed P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260925 03:04:44.310297 26293 raft_consensus.cc:729] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [term 4 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:44.271248 26414 raft_consensus.cc:504] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:44.443995 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be4e0aa984f4f2b81a58e382e434ad5" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:44.445781 24437 leader_election.cc:302] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:44.395499 26518 raft_consensus.cc:482] T 8ac466793d094325ace224708b2158ed P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:44.394731 26376 raft_consensus.cc:686] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 LEADER]: Becoming Leader. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:04:44.527436 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b030d84c670b48d5bff337ccc0c77626" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:44.434302 26372 leader_election.cc:288] T 420163a6cdce412ebc21c25fe3835638 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:44.439610 26451 raft_consensus.cc:482] T 0b5df303cd6a4d59aca5c18bbbff24ed P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:44.458555 26363 raft_consensus.cc:482] T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:44.446655 26438 raft_consensus.cc:482] T 43366a0a2f9d4344bf4e37ffd290ef8a P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:44.458228 26439 raft_consensus.cc:482] T 71f461a6e2ff434993b78a2d077f0abd P 1bc1f2e62ab44af995165bd5116a9d43 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:44.407577 26499 raft_consensus.cc:504] T 9e8af2641e124f9589c5ccfdac785f63 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:44.462770 26497 raft_consensus.cc:3032] T 7993f9c3e2a44f408bbf5d1e395e2306 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 LEADER]: Stepping down as leader of term 4
I20260925 03:04:44.456604 26412 raft_consensus.cc:482] T 8e5819c15eac4544a57e5ee6c9d4f117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:44.568499 26325 raft_consensus.cc:2726] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260925 03:04:44.634207 26451 raft_consensus.cc:3037] T 0b5df303cd6a4d59aca5c18bbbff24ed P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:04:44.690368 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367277daf8d04778b3fa4ed94d71580d" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:44.713858 24514 leader_election.cc:302] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:44.714140 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:44.714972 24464 raft_consensus.cc:2454] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 5.
I20260925 03:04:44.716787 24514 leader_election.cc:302] T b4ebd2b043024a548005c8b6b00b0449 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:04:44.718146 24514 leader_election.cc:302] T 48ca6ccd98194cf2af7b5278d15bf4ea P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:44.719367 24514 leader_election.cc:302] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:44.630213 26362 raft_consensus.cc:482] T a4abe0ea757e4886be4945c5001e529a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:44.738194 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4abe0ea757e4886be4945c5001e529a" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
W20260925 03:04:44.740963 26531 log.cc:924] Time spent T 550118bc1a0641ce9803bdbf43f2069f P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.435s user 0.000s sys 0.000s
I20260925 03:04:44.641808 26379 raft_consensus.cc:2726] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260925 03:04:44.758607 24471 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 938) took 1339 ms. Trace:
I20260925 03:04:44.649966 25979 raft_consensus.cc:686] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [term 6 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:44.652001 26406 raft_consensus.cc:482] T 3d89396576884f7c8901dba54f783947 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:44.765286 24471 rpcz_store.cc:276] 0925 03:04:43.419607 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:43.424673 (+ 5066us) service_pool.cc:227] Handling call
0925 03:04:44.758409 (+1333736us) spinlock_profiling.cc:243] Waited 285 ms on lock 0x7b54003eb7d4. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d42ae 00007f437b411c27 00007f4374af9c04 00007f4374af9b92 00007f4374af9ac1 00007f4374af9a3c 00007f4374af8cf2 00007f43748fa94c 00007f43748f9d86 00007f43748f972f
0925 03:04:44.758550 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":656622848}
I20260925 03:04:44.741479 26522 raft_consensus.cc:3032] T b1e85248d3b04b979dd080caeeb02fdb P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 LEADER]: Stepping down as leader of term 4
I20260925 03:04:44.770898 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6b6ec010caf4689b8c90449764f5cf9" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:44.771974 24467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5f9b503b0ab43999d582015a367441e" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:44.774551 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367277daf8d04778b3fa4ed94d71580d" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:44.671358 26497 raft_consensus.cc:729] T 7993f9c3e2a44f408bbf5d1e395e2306 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:04:44.783102 26522 raft_consensus.cc:729] T b1e85248d3b04b979dd080caeeb02fdb P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1bc1f2e62ab44af995165bd5116a9d43, State: Running, Role: LEADER
I20260925 03:04:44.790794 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e92f377a79354a6ebb4eec67303c13e3" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:44.726320 26368 raft_consensus.cc:2726] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260925 03:04:44.626379 26518 raft_consensus.cc:504] T 8ac466793d094325ace224708b2158ed P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
W20260925 03:04:44.772238 26537 log.cc:924] Time spent T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.073s user 0.000s sys 0.000s
I20260925 03:04:44.801348 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "293ef69caf514ac98510a68495f54eda" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:44.462613 26507 consensus_queue.cc:237] T edcf6e6a422d4d198d4fbb0f7d3d845d P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:44.659219 26363 raft_consensus.cc:504] T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:44.674811 26438 raft_consensus.cc:504] T 43366a0a2f9d4344bf4e37ffd290ef8a P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:44.688750 26439 raft_consensus.cc:504] T 71f461a6e2ff434993b78a2d077f0abd P 1bc1f2e62ab44af995165bd5116a9d43 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:44.829460 24548 raft_consensus.cc:2454] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 4.
I20260925 03:04:44.730222 26413 consensus_queue.cc:237] T a7aa47c48404465b94a36fd819fe3c7a P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:04:44.835201 26525 log.cc:924] Time spent T 442d4a29b1d743599f4b6b4871e26668 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.558s user 0.000s sys 0.001s
I20260925 03:04:44.836313 24542 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 1339) took 1176 ms. Trace:
I20260925 03:04:44.777796 26262 consensus_queue.cc:237] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:44.837060 26532 raft_consensus.cc:482] T 11bb46bce3564841b7674e6a693bd2e6 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
W20260925 03:04:44.837529 26528 log.cc:924] Time spent T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.393s user 0.000s sys 0.001s
I20260925 03:04:44.838224 26509 raft_consensus.cc:3032] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [term 4 LEADER]: Stepping down as leader of term 4
I20260925 03:04:44.838652 26509 raft_consensus.cc:729] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [term 4 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:44.836830 24542 rpcz_store.cc:276] 0925 03:04:43.660231 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:43.704612 (+ 44381us) service_pool.cc:227] Handling call
0925 03:04:43.704979 (+ 367us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:04:43.947591 (+242612us) raft_consensus.cc:1520] Early marking committed up to index 2
0925 03:04:43.947623 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:04:43.948032 (+ 409us) log.cc:842] Serialized 26 byte log entry
0925 03:04:44.114438 (+166406us) raft_consensus.cc:1631] Marking committed up to 2
0925 03:04:44.114477 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:44.114546 (+ 69us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:04:44.836076 (+721530us) raft_consensus.cc:1671] finished
0925 03:04:44.836103 (+ 27us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:44.836250 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":231957,"raft.queue_time_us":237329,"raft.run_cpu_time_us":94,"raft.run_wall_time_us":95,"thread_start_us":166012,"threads_started":1,"wal-append.queue_time_us":253066}
I20260925 03:04:44.840502 24546 raft_consensus.cc:2454] T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 4.
I20260925 03:04:44.841032 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:44.833542 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15861cd28e354598a3c6bfc4337c34bb" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:44.823179 26553 raft_consensus.cc:2726] T b4ebd2b043024a548005c8b6b00b0449 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260925 03:04:44.842233 24542 raft_consensus.cc:2362] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 5. Re-sending same reply.
I20260925 03:04:44.704437 26293 consensus_queue.cc:260] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:44.773232 26412 raft_consensus.cc:504] T 8e5819c15eac4544a57e5ee6c9d4f117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:04:44.822494 26383 consensus_meta.cc:287] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c: Time spent flushing consensus metadata: real 0.762s user 0.010s sys 0.001s
I20260925 03:04:44.773600 26362 raft_consensus.cc:504] T a4abe0ea757e4886be4945c5001e529a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:44.793234 26406 raft_consensus.cc:504] T 3d89396576884f7c8901dba54f783947 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:44.823662 26368 raft_consensus.cc:2726] T 48ca6ccd98194cf2af7b5278d15bf4ea P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260925 03:04:44.851589 26509 consensus_queue.cc:260] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:04:44.852810 26516 log.cc:924] Time spent T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.099s user 0.000s sys 0.001s
W20260925 03:04:44.853001 26535 log.cc:924] Time spent T e6b6ec010caf4689b8c90449764f5cf9 P 1bc1f2e62ab44af995165bd5116a9d43: Append to log took a long time: real 0.075s user 0.000s sys 0.001s
I20260925 03:04:44.853267 26412 leader_election.cc:288] T 8e5819c15eac4544a57e5ee6c9d4f117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:44.851599 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "293ef69caf514ac98510a68495f54eda" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:44.881228 24542 raft_consensus.cc:2362] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 4. Re-sending same reply.
I20260925 03:04:44.881332 24545 raft_consensus.cc:2454] T a4abe0ea757e4886be4945c5001e529a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 2.
I20260925 03:04:44.882249 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15861cd28e354598a3c6bfc4337c34bb" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:44.883421 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43366a0a2f9d4344bf4e37ffd290ef8a" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:44.884193 24542 raft_consensus.cc:2396] T 43366a0a2f9d4344bf4e37ffd290ef8a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260925 03:04:44.856112 26438 leader_election.cc:288] T 43366a0a2f9d4344bf4e37ffd290ef8a P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:44.856752 26553 raft_consensus.cc:2726] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260925 03:04:44.858290 26439 leader_election.cc:288] T 71f461a6e2ff434993b78a2d077f0abd P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:44.859627 26499 leader_election.cc:288] T 9e8af2641e124f9589c5ccfdac785f63 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:44.862113 26357 raft_consensus.cc:482] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:44.885963 26357 raft_consensus.cc:504] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:44.889914 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6b6ec010caf4689b8c90449764f5cf9" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:44.890803 24542 raft_consensus.cc:2454] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 4.
I20260925 03:04:44.891752 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:44.824160 26522 consensus_queue.cc:260] T b1e85248d3b04b979dd080caeeb02fdb P 1bc1f2e62ab44af995165bd5116a9d43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
W20260925 03:04:44.836943 26524 log.cc:924] Time spent T 5278a60f97da4e10aecd50a8d9a9a43c P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.125s user 0.000s sys 0.001s
I20260925 03:04:44.830657 26530 raft_consensus.cc:482] T 100c113a9fd84cc8adf2ec7b33ea0093 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
W20260925 03:04:44.847901 26527 log.cc:924] Time spent T cc2df40db08649c9a75e90000b878559 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.570s user 0.001s sys 0.000s
I20260925 03:04:44.783994 26376 consensus_queue.cc:237] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:44.892179 24614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71f461a6e2ff434993b78a2d077f0abd" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:44.893131 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ac466793d094325ace224708b2158ed" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
W20260925 03:04:44.893319 26540 log.cc:924] Time spent T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.190s user 0.000s sys 0.001s
W20260925 03:04:44.893898 26536 log.cc:924] Time spent T 420163a6cdce412ebc21c25fe3835638 P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.450s user 0.001s sys 0.000s
W20260925 03:04:44.894131 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 118239us
I20260925 03:04:44.923273 24586 leader_election.cc:302] T 43366a0a2f9d4344bf4e37ffd290ef8a P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5
W20260925 03:04:44.924216 26386 consensus_meta.cc:287] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c: Time spent flushing consensus metadata: real 0.759s user 0.010s sys 0.000s
I20260925 03:04:44.924647 26386 raft_consensus.cc:504] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:44.895440 26363 leader_election.cc:288] T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:44.926304 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "420163a6cdce412ebc21c25fe3835638" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:44.926946 24543 raft_consensus.cc:2454] T 420163a6cdce412ebc21c25fe3835638 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 5.
I20260925 03:04:44.896947 26362 leader_election.cc:288] T a4abe0ea757e4886be4945c5001e529a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:44.865832 26518 leader_election.cc:288] T 8ac466793d094325ace224708b2158ed P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:44.899101 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e8af2641e124f9589c5ccfdac785f63" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:44.941506 24541 raft_consensus.cc:2379] T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 in current term 4: Already voted for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in this term.
I20260925 03:04:44.899559 26406 leader_election.cc:288] T 3d89396576884f7c8901dba54f783947 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:44.836221 26497 consensus_queue.cc:260] T 7993f9c3e2a44f408bbf5d1e395e2306 P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:44.898520 26270 leader_election.cc:288] T 25102ba08e4e49388f6176badcd10a25 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
W20260925 03:04:44.892589 26373 consensus_meta.cc:287] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent flushing consensus metadata: real 0.649s user 0.005s sys 0.002s
I20260925 03:04:44.901455 24542 raft_consensus.cc:2454] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 5.
I20260925 03:04:44.876118 26421 leader_election.cc:288] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:44.919508 24614 raft_consensus.cc:2454] T 71f461a6e2ff434993b78a2d077f0abd P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 5.
I20260925 03:04:44.921311 26553 raft_consensus.cc:482] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:44.950716 24542 raft_consensus.cc:1226] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 4. Current term is 5. Ops: []
I20260925 03:04:44.927062 26386 leader_election.cc:288] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:44.837598 26532 raft_consensus.cc:504] T 11bb46bce3564841b7674e6a693bd2e6 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:44.944842 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25102ba08e4e49388f6176badcd10a25" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:44.868165 25979 consensus_queue.cc:237] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:04:44.948235 25165 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.500s user 0.002s sys 0.006s
I20260925 03:04:44.928920 24617 raft_consensus.cc:1226] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 4. Current term is 5. Ops: []
I20260925 03:04:44.933465 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ac285175464fbda67211d9574db415" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:44.972079 24544 raft_consensus.cc:2454] T 25102ba08e4e49388f6176badcd10a25 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 2.
I20260925 03:04:44.938997 26363 raft_consensus.cc:482] T 0b5df303cd6a4d59aca5c18bbbff24ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
W20260925 03:04:44.973241 25165 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.525s user 0.002s sys 0.006s
W20260925 03:04:44.979314 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 48770.1us
I20260925 03:04:44.980043 24586 leader_election.cc:302] T 420163a6cdce412ebc21c25fe3835638 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:04:44.974807 24548 raft_consensus.cc:3037] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:44.949980 26366 raft_consensus.cc:482] T 71f461a6e2ff434993b78a2d077f0abd P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:44.957715 26497 raft_consensus.cc:3037] T 7993f9c3e2a44f408bbf5d1e395e2306 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:44.945992 26246 raft_consensus.cc:2781] T 43366a0a2f9d4344bf4e37ffd290ef8a P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260925 03:04:44.994082 24465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25102ba08e4e49388f6176badcd10a25" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:44.995532 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ac285175464fbda67211d9574db415" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:44.995924 24465 raft_consensus.cc:2454] T 25102ba08e4e49388f6176badcd10a25 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 2.
I20260925 03:04:44.944139 26451 raft_consensus.cc:504] T 0b5df303cd6a4d59aca5c18bbbff24ed P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:04:44.981545 24539 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.757s user 0.009s sys 0.005s
I20260925 03:04:45.020995 24586 leader_election.cc:302] T 25102ba08e4e49388f6176badcd10a25 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:45.034497 24548 raft_consensus.cc:2454] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 5.
I20260925 03:04:44.980834 26270 raft_consensus.cc:482] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:44.984249 26363 raft_consensus.cc:504] T 0b5df303cd6a4d59aca5c18bbbff24ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:45.030390 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e5819c15eac4544a57e5ee6c9d4f117" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:45.023797 26262 raft_consensus.cc:2781] T 420163a6cdce412ebc21c25fe3835638 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260925 03:04:44.949606 26373 raft_consensus.cc:504] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:44.924477 26530 raft_consensus.cc:504] T 100c113a9fd84cc8adf2ec7b33ea0093 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:45.041036 24586 leader_election.cc:302] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
W20260925 03:04:45.034384 24539 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.818s user 0.010s sys 0.006s
I20260925 03:04:45.056470 24621 raft_consensus.cc:1226] T a7aa47c48404465b94a36fd819fe3c7a P eeab93abfdb141a895002ec9f312296c [term 4 LEADER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 3. Current term is 4. Ops: []
I20260925 03:04:44.999837 26246 raft_consensus.cc:482] T 43366a0a2f9d4344bf4e37ffd290ef8a P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Starting leader election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:44.964195 26553 raft_consensus.cc:504] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:44.988775 26406 consensus_queue.cc:1046] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.058s
I20260925 03:04:44.989100 25162 leader_election.cc:302] T 71f461a6e2ff434993b78a2d077f0abd P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:45.039737 26262 raft_consensus.cc:482] T 420163a6cdce412ebc21c25fe3835638 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Starting leader election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:44.998049 26366 raft_consensus.cc:504] T 71f461a6e2ff434993b78a2d077f0abd P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:45.070082 26439 raft_consensus.cc:2781] T 71f461a6e2ff434993b78a2d077f0abd P 1bc1f2e62ab44af995165bd5116a9d43 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260925 03:04:45.070510 26439 raft_consensus.cc:482] T 71f461a6e2ff434993b78a2d077f0abd P 1bc1f2e62ab44af995165bd5116a9d43 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:45.070840 26439 raft_consensus.cc:3037] T 71f461a6e2ff434993b78a2d077f0abd P 1bc1f2e62ab44af995165bd5116a9d43 [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:04:45.058224 26448 raft_consensus.cc:2781] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Leader election won for term 5
I20260925 03:04:45.060941 26551 raft_consensus.cc:2781] T 25102ba08e4e49388f6176badcd10a25 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260925 03:04:45.077044 26551 raft_consensus.cc:482] T 25102ba08e4e49388f6176badcd10a25 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:45.077392 26551 raft_consensus.cc:3037] T 25102ba08e4e49388f6176badcd10a25 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:45.079701 24540 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 847) took 1070 ms. Trace:
I20260925 03:04:45.038625 26270 raft_consensus.cc:504] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:45.080094 24540 rpcz_store.cc:276] 0925 03:04:44.008965 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:44.424498 (+415533us) service_pool.cc:227] Handling call
0925 03:04:44.449589 (+ 25091us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:04:44.537156 (+ 87567us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:44.537192 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:04:44.539687 (+ 2495us) log.cc:842] Serialized 26 byte log entry
0925 03:04:44.998758 (+459071us) raft_consensus.cc:1631] Marking committed up to 3
0925 03:04:45.031318 (+ 32560us) log.cc:842] Serialized 14 byte log entry
0925 03:04:45.031510 (+ 192us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:45.031580 (+ 70us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:04:45.079457 (+ 47877us) raft_consensus.cc:1671] finished
0925 03:04:45.079478 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:45.079641 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":39163,"prepare.queue_time_us":32631,"prepare.run_cpu_time_us":29,"prepare.run_wall_time_us":29,"raft.queue_time_us":70378,"raft.run_cpu_time_us":280,"raft.run_wall_time_us":49222,"thread_start_us":241541,"threads_started":1,"wal-append.queue_time_us":405357}
I20260925 03:04:45.083937 24548 raft_consensus.cc:1261] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260925 03:04:45.069185 26262 raft_consensus.cc:3037] T 420163a6cdce412ebc21c25fe3835638 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:04:45.060274 26246 raft_consensus.cc:3037] T 43366a0a2f9d4344bf4e37ffd290ef8a P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Advancing to term 3
W20260925 03:04:45.082051 26427 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.770s user 0.011s sys 0.011s
W20260925 03:04:45.089581 24547 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.853s user 0.005s sys 0.018s
I20260925 03:04:45.062224 26413 raft_consensus.cc:482] T 48ca6ccd98194cf2af7b5278d15bf4ea P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260925 03:04:45.097363 24547 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.861s user 0.006s sys 0.018s
I20260925 03:04:45.098767 25198 raft_consensus.cc:1261] T 29637f5a4a504ce6ba29033ccd0cd716 P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260925 03:04:45.092907 24542 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 853) took 1015 ms. Trace:
I20260925 03:04:45.094502 24548 raft_consensus.cc:1226] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260925 03:04:45.094859 24620 raft_consensus.cc:2454] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 4.
I20260925 03:04:45.080921 26363 leader_election.cc:288] T 0b5df303cd6a4d59aca5c18bbbff24ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
W20260925 03:04:45.073776 25162 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 94486.1us
W20260925 03:04:45.097038 26427 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.791s user 0.012s sys 0.012s
I20260925 03:04:45.112092 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
W20260925 03:04:45.113265 25163 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.682s user 0.017s sys 0.017s
W20260925 03:04:45.113613 25163 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.682s user 0.017s sys 0.017s
I20260925 03:04:45.115401 24622 raft_consensus.cc:2454] T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 5.
I20260925 03:04:45.119664 24541 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 850) took 1061 ms. Trace:
I20260925 03:04:45.121466 24471 raft_consensus.cc:1261] T 550118bc1a0641ce9803bdbf43f2069f P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260925 03:04:45.108071 26539 log.cc:924] Time spent T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5: Append to log took a long time: real 0.117s user 0.001s sys 0.000s
I20260925 03:04:45.092098 26448 raft_consensus.cc:686] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [term 5 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
W20260925 03:04:45.125931 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 406613us
I20260925 03:04:45.129348 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43366a0a2f9d4344bf4e37ffd290ef8a" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
W20260925 03:04:45.130390 25163 leader_election.cc:334] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389): Timed out: RequestConsensusVote RPC to 127.22.193.2:36389 timed out after 1.735s (SENT)
I20260925 03:04:45.132062 24616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e8af2641e124f9589c5ccfdac785f63" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:45.110258 24539 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 839) took 1138 ms. Trace:
I20260925 03:04:45.111615 26427 raft_consensus.cc:686] T 7993f9c3e2a44f408bbf5d1e395e2306 P eeab93abfdb141a895002ec9f312296c [term 5 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:45.131479 24514 leader_election.cc:302] T c85f5d1d3f004cf5817bb594fd8e6469 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:45.150010 24467 raft_consensus.cc:1261] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Refusing update from remote peer 1bc1f2e62ab44af995165bd5116a9d43: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260925 03:04:45.125180 26373 leader_election.cc:288] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 6 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:45.154598 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b5df303cd6a4d59aca5c18bbbff24ed" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:45.155259 24469 raft_consensus.cc:3037] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:04:45.128014 25191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:45.170056 24466 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 978) took 1023 ms. Trace:
I20260925 03:04:45.157971 24514 leader_election.cc:302] T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:45.152545 25193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6b6ec010caf4689b8c90449764f5cf9" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
W20260925 03:04:45.166913 26560 log.cc:924] Time spent T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.070s user 0.000s sys 0.000s
I20260925 03:04:45.167472 26553 leader_election.cc:288] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:45.168768 25190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:45.172957 24466 rpcz_store.cc:276] 0925 03:04:44.146672 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:44.146781 (+ 109us) service_pool.cc:227] Handling call
0925 03:04:44.147070 (+ 289us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:04:44.151580 (+ 4510us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:44.151616 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:04:44.152100 (+ 484us) log.cc:842] Serialized 26 byte log entry
0925 03:04:44.235998 (+ 83898us) raft_consensus.cc:1631] Marking committed up to 1
0925 03:04:44.236035 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:44.236499 (+ 464us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:04:45.169821 (+933322us) raft_consensus.cc:1671] finished
0925 03:04:45.169842 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:45.169991 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":106,"raft.run_cpu_time_us":127,"raft.run_wall_time_us":5708,"thread_start_us":35112,"threads_started":1,"wal-append.queue_time_us":767099}
I20260925 03:04:45.174363 24514 leader_election.cc:302] T 550118bc1a0641ce9803bdbf43f2069f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:04:45.176854 24616 raft_consensus.cc:2454] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 4.
I20260925 03:04:45.112267 26413 raft_consensus.cc:504] T 48ca6ccd98194cf2af7b5278d15bf4ea P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:45.151291 26406 raft_consensus.cc:2726] T c85f5d1d3f004cf5817bb594fd8e6469 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260925 03:04:45.141470 26383 consensus_queue.cc:1035] T 550118bc1a0641ce9803bdbf43f2069f P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.050s
I20260925 03:04:45.198961 24514 leader_election.cc:302] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:04:45.195205 24469 raft_consensus.cc:2454] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 6.
I20260925 03:04:45.208374 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e5819c15eac4544a57e5ee6c9d4f117" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:45.210661 24588 leader_election.cc:302] T 0b5df303cd6a4d59aca5c18bbbff24ed P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:45.190692 24544 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 856) took 1112 ms. Trace:
I20260925 03:04:45.209263 24469 raft_consensus.cc:2454] T 8e5819c15eac4544a57e5ee6c9d4f117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 4.
I20260925 03:04:45.203783 26553 raft_consensus.cc:2741] T 550118bc1a0641ce9803bdbf43f2069f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260925 03:04:45.189711 26571 raft_consensus.cc:2726] T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260925 03:04:45.192795 26519 log.cc:924] Time spent T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5: Append to log took a long time: real 1.215s user 0.003s sys 0.015s
I20260925 03:04:45.220911 24588 leader_election.cc:302] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:45.223387 26549 raft_consensus.cc:2781] T 0b5df303cd6a4d59aca5c18bbbff24ed P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Leader election won for term 6
I20260925 03:04:45.226991 26439 raft_consensus.cc:504] T 71f461a6e2ff434993b78a2d077f0abd P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:45.229496 25198 raft_consensus.cc:1226] T 71f461a6e2ff434993b78a2d077f0abd P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20260925 03:04:45.190215 26377 consensus_queue.cc:1035] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.028s
I20260925 03:04:45.194839 26530 leader_election.cc:288] T 100c113a9fd84cc8adf2ec7b33ea0093 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:45.235489 26270 leader_election.cc:288] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:45.230674 24616 raft_consensus.cc:1261] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Refusing update from remote peer 1bc1f2e62ab44af995165bd5116a9d43: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260925 03:04:45.200410 26451 leader_election.cc:288] T 0b5df303cd6a4d59aca5c18bbbff24ed P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 6 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:45.205051 26448 consensus_queue.cc:237] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:45.193349 26551 raft_consensus.cc:504] T 25102ba08e4e49388f6176badcd10a25 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:45.192981 25191 raft_consensus.cc:2454] T 7e49248643df4131a1c42e74244e05c2 P 1bc1f2e62ab44af995165bd5116a9d43 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 5.
I20260925 03:04:45.215263 26427 consensus_queue.cc:237] T 7993f9c3e2a44f408bbf5d1e395e2306 P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:45.242936 26439 leader_election.cc:288] T 71f461a6e2ff434993b78a2d077f0abd P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 6 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:45.245635 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b5df303cd6a4d59aca5c18bbbff24ed" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:45.250124 24548 raft_consensus.cc:3037] T 0b5df303cd6a4d59aca5c18bbbff24ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Advancing to term 6
I20260925 03:04:45.241539 26247 raft_consensus.cc:2781] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260925 03:04:45.240547 26549 raft_consensus.cc:686] T 0b5df303cd6a4d59aca5c18bbbff24ed P eeab93abfdb141a895002ec9f312296c [term 6 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:45.120100 24541 rpcz_store.cc:276] 0925 03:04:44.058391 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:44.942145 (+883754us) service_pool.cc:227] Handling call
0925 03:04:44.942470 (+ 325us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:44.942673 (+ 203us) raft_consensus.cc:1520] Early marking committed up to index 3
0925 03:04:44.948769 (+ 6096us) log.cc:842] Serialized 14 byte log entry
0925 03:04:45.119375 (+170606us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:45.119416 (+ 41us) raft_consensus.cc:1631] Marking committed up to 3
0925 03:04:45.119444 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:45.119488 (+ 44us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:45.119566 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1708,"prepare.queue_time_us":5906,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":49,"raft.queue_time_us":3032,"thread_start_us":168498,"threads_started":1}
I20260925 03:04:45.258522 26247 raft_consensus.cc:482] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:45.271795 24541 raft_consensus.cc:1226] T 7993f9c3e2a44f408bbf5d1e395e2306 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 4. Current term is 5. Ops: []
I20260925 03:04:45.272068 26247 raft_consensus.cc:3037] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:45.272507 24541 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 1357) took 1022 ms. Trace:
I20260925 03:04:45.272873 24541 rpcz_store.cc:276] 0925 03:04:44.250180 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:45.271320 (+1021140us) service_pool.cc:227] Handling call
0925 03:04:45.271631 (+ 311us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:45.272270 (+ 639us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:45.272456 (+ 186us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:45.136452 24539 rpcz_store.cc:276] 0925 03:04:43.961917 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:43.970188 (+ 8271us) service_pool.cc:227] Handling call
0925 03:04:43.980543 (+ 10355us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:04:43.981021 (+ 478us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:43.981059 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:04:43.981537 (+ 478us) log.cc:842] Serialized 26 byte log entry
0925 03:04:44.128539 (+147002us) raft_consensus.cc:1631] Marking committed up to 3
0925 03:04:45.064482 (+935943us) log.cc:842] Serialized 14 byte log entry
0925 03:04:45.082436 (+ 17954us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:45.088740 (+ 6304us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:04:45.089985 (+ 1245us) raft_consensus.cc:1671] finished
0925 03:04:45.090001 (+ 16us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:45.100753 (+ 10752us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":28066,"prepare.queue_time_us":135749,"prepare.run_cpu_time_us":70,"prepare.run_wall_time_us":10030,"raft.queue_time_us":10275,"raft.run_cpu_time_us":335,"raft.run_wall_time_us":1879,"thread_start_us":989246,"threads_started":2,"wal-append.queue_time_us":173023}
I20260925 03:04:45.279820 24616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71f461a6e2ff434993b78a2d077f0abd" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:45.262143 26439 consensus_queue.cc:1035] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.020s
W20260925 03:04:45.282502 25163 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.22.193.2:36389 timed out after 1.516s (SENT) [suppressed 29 similar messages]
I20260925 03:04:45.324544 24548 raft_consensus.cc:2454] T 0b5df303cd6a4d59aca5c18bbbff24ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 6.
I20260925 03:04:45.345041 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e92f377a79354a6ebb4eec67303c13e3" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:45.345095 24616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f68804eb5a244d87bb395d50683c6012" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:45.353350 24548 raft_consensus.cc:1226] T a7aa47c48404465b94a36fd819fe3c7a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 3. Current term is 4. Ops: []
W20260925 03:04:45.363618 24436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 170035us
I20260925 03:04:45.367089 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4abe0ea757e4886be4945c5001e529a" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:45.367560 24436 leader_election.cc:302] T 43366a0a2f9d4344bf4e37ffd290ef8a P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:45.338567 26357 leader_election.cc:288] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:45.372203 24436 leader_election.cc:302] T 9e8af2641e124f9589c5ccfdac785f63 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:45.374258 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d89396576884f7c8901dba54f783947" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:45.376259 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b5df303cd6a4d59aca5c18bbbff24ed" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:45.377444 24436 leader_election.cc:302] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
W20260925 03:04:45.392076 25163 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 109595us
W20260925 03:04:45.393278 25163 leader_election.cc:334] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389): Timed out: RequestConsensusVote RPC to 127.22.193.2:36389 timed out after 1.673s (SENT)
I20260925 03:04:45.393743 25163 leader_election.cc:302] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:45.373567 26437 raft_consensus.cc:2726] T 43366a0a2f9d4344bf4e37ffd290ef8a P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260925 03:04:45.399120 24615 raft_consensus.cc:1226] T 7993f9c3e2a44f408bbf5d1e395e2306 P eeab93abfdb141a895002ec9f312296c [term 5 LEADER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 4. Current term is 5. Ops: [4.2-4.2]
I20260925 03:04:45.392336 24539 raft_consensus.cc:1226] T b4ebd2b043024a548005c8b6b00b0449 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Rejecting Update request from peer 1bc1f2e62ab44af995165bd5116a9d43 for earlier term 2. Current term is 3. Ops: []
I20260925 03:04:45.395747 26557 raft_consensus.cc:2726] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260925 03:04:45.396656 26413 leader_election.cc:288] T 48ca6ccd98194cf2af7b5278d15bf4ea P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:45.396888 26580 raft_consensus.cc:482] T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1bc1f2e62ab44af995165bd5116a9d43)
I20260925 03:04:45.412886 25193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e92f377a79354a6ebb4eec67303c13e3" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:45.415551 26437 raft_consensus.cc:2781] T 9e8af2641e124f9589c5ccfdac785f63 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader election won for term 4
I20260925 03:04:45.393975 26554 consensus_queue.cc:1035] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.020s
W20260925 03:04:45.417477 26572 log.cc:924] Time spent T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.098s user 0.000s sys 0.000s
I20260925 03:04:45.417620 26246 raft_consensus.cc:504] T 43366a0a2f9d4344bf4e37ffd290ef8a P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:04:45.420305 26575 log.cc:924] Time spent T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.176s user 0.001s sys 0.000s
I20260925 03:04:45.421922 24540 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 1356) took 1172 ms. Trace:
I20260925 03:04:45.439853 24540 rpcz_store.cc:276] 0925 03:04:44.249923 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:45.242759 (+992836us) service_pool.cc:227] Handling call
0925 03:04:45.243083 (+ 324us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:04:45.243279 (+ 196us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:45.243301 (+ 22us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:04:45.243674 (+ 373us) log.cc:842] Serialized 26 byte log entry
0925 03:04:45.341125 (+ 97451us) raft_consensus.cc:1631] Marking committed up to 2
0925 03:04:45.341160 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:45.341223 (+ 63us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:04:45.421725 (+ 80502us) raft_consensus.cc:1671] finished
0925 03:04:45.421743 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:45.421872 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":97104,"threads_started":1,"wal-append.queue_time_us":99994}
I20260925 03:04:45.413399 26551 leader_election.cc:288] T 25102ba08e4e49388f6176badcd10a25 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 3 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:45.427932 26438 raft_consensus.cc:2781] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260925 03:04:45.445722 25199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4abe0ea757e4886be4945c5001e529a" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:45.454401 26247 raft_consensus.cc:504] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:45.460426 26386 raft_consensus.cc:482] T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:45.382061 26549 consensus_queue.cc:237] T 0b5df303cd6a4d59aca5c18bbbff24ed P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:45.464116 26246 leader_election.cc:288] T 43366a0a2f9d4344bf4e37ffd290ef8a P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 3 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:45.472617 24511 leader_election.cc:302] T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 1bc1f2e62ab44af995165bd5116a9d43, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:45.471479 26437 raft_consensus.cc:686] T 9e8af2641e124f9589c5ccfdac785f63 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 LEADER]: Becoming Leader. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:04:45.483786 24540 raft_consensus.cc:1261] T 420163a6cdce412ebc21c25fe3835638 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260925 03:04:45.491129 24470 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:55552 (request call id 445) took 1273 ms. Trace:
I20260925 03:04:45.473404 26438 raft_consensus.cc:482] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:45.491909 26438 raft_consensus.cc:3037] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:04:45.493036 26437 consensus_queue.cc:237] T 9e8af2641e124f9589c5ccfdac785f63 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:45.214695 24544 rpcz_store.cc:276] 0925 03:04:44.078210 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:44.974578 (+896368us) service_pool.cc:227] Handling call
0925 03:04:44.992159 (+ 17581us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:44.992341 (+ 182us) raft_consensus.cc:1520] Early marking committed up to index 3
0925 03:04:45.030229 (+ 37888us) log.cc:842] Serialized 14 byte log entry
0925 03:04:45.190273 (+160044us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:45.190355 (+ 82us) raft_consensus.cc:1631] Marking committed up to 3
0925 03:04:45.190422 (+ 67us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:45.190493 (+ 71us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:45.190623 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":29209,"prepare.queue_time_us":38838,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":33,"raft.queue_time_us":29349,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":3606,"spinlock_wait_cycles":22045952,"thread_start_us":130329,"threads_started":1,"wal-append.queue_time_us":162109}
I20260925 03:04:45.491601 24470 rpcz_store.cc:276] 0925 03:04:44.217318 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:44.217498 (+ 180us) service_pool.cc:227] Handling call
0925 03:04:45.490456 (+1272958us) spinlock_profiling.cc:243] Waited 42.7 ms on lock 0x7b5400141254. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455af0 00007f43764d8f2a 00007f437b415c6e 00007f4374b0c807 00007f4374b0c792 00007f4374b0c6c1 00007f4374b0c63c 00007f4374b0b8f2 00007f43748fa94c 00007f43748f9d86 00007f43748f972f
0925 03:04:45.491022 (+ 566us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":98264448}
I20260925 03:04:45.497786 26532 leader_election.cc:288] T 11bb46bce3564841b7674e6a693bd2e6 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:45.500572 24544 raft_consensus.cc:1261] T 550118bc1a0641ce9803bdbf43f2069f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260925 03:04:45.507630 24617 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38570 (request call id 1394) took 1215 ms. Trace:
I20260925 03:04:45.508018 24617 rpcz_store.cc:276] 0925 03:04:44.289158 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:44.289525 (+ 367us) service_pool.cc:227] Handling call
0925 03:04:44.289831 (+ 306us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:44.973359 (+683528us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:45.504430 (+531071us) spinlock_profiling.cc:243] Waited 639 ms on lock 0x7b540018f6d4. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d0f51 00007f43764cf99e 00007f437b4112e9 00007f4374af7684 00007f4374af7612 00007f4374af7541 00007f4374af74bc 00007f4374af6772 00007f43748fa94c 00007f43748f9d86
0925 03:04:45.504628 (+ 198us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1469377408}
I20260925 03:04:45.509951 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7691c6b475428c82d6181d693d26f3" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:45.510329 26247 leader_election.cc:288] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:45.487146 26386 raft_consensus.cc:504] T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:45.431699 26580 raft_consensus.cc:504] T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:45.506305 26262 raft_consensus.cc:504] T 420163a6cdce412ebc21c25fe3835638 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:45.572590 24619 raft_consensus.cc:1226] T 420163a6cdce412ebc21c25fe3835638 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 5. Current term is 6. Ops: []
I20260925 03:04:45.593677 24464 raft_consensus.cc:3032] T a7aa47c48404465b94a36fd819fe3c7a P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 LEADER]: Stepping down as leader of term 3
I20260925 03:04:45.594161 24464 raft_consensus.cc:729] T a7aa47c48404465b94a36fd819fe3c7a P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:04:45.594879 24464 consensus_queue.cc:260] T a7aa47c48404465b94a36fd819fe3c7a P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:45.604457 24465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43366a0a2f9d4344bf4e37ffd290ef8a" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:45.604842 24464 raft_consensus.cc:3037] T a7aa47c48404465b94a36fd819fe3c7a P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:45.605516 24465 raft_consensus.cc:2379] T 43366a0a2f9d4344bf4e37ffd290ef8a P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c in current term 3: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:04:45.562103 26386 leader_election.cc:288] T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
W20260925 03:04:45.563979 26400 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.045s user 0.000s sys 0.022s
I20260925 03:04:45.615819 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48ca6ccd98194cf2af7b5278d15bf4ea" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:45.616477 24470 raft_consensus.cc:2454] T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 3.
I20260925 03:04:45.618198 24472 raft_consensus.cc:1261] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 3. (term mismatch)
I20260925 03:04:45.618548 24472 pending_rounds.cc:76] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5: Aborting all ops after (but not including) 2
I20260925 03:04:45.618741 24472 pending_rounds.cc:97] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20260925 03:04:45.618953 24472 raft_consensus.cc:2866] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260925 03:04:45.570178 26507 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.711s user 0.014s sys 0.028s
I20260925 03:04:45.617125 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25102ba08e4e49388f6176badcd10a25" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:45.622184 24473 raft_consensus.cc:3037] T 25102ba08e4e49388f6176badcd10a25 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Advancing to term 3
W20260925 03:04:45.613598 26400 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.154s user 0.000s sys 0.023s
W20260925 03:04:45.623172 26507 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.764s user 0.014s sys 0.029s
W20260925 03:04:45.628182 24436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.4:43979, user_credentials={real_user=slave}} blocked reactor thread for 250805us
I20260925 03:04:45.628317 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e5819c15eac4544a57e5ee6c9d4f117" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:45.620879 26554 raft_consensus.cc:482] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:45.628950 24466 raft_consensus.cc:3037] T 8e5819c15eac4544a57e5ee6c9d4f117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:45.630061 24464 raft_consensus.cc:1261] T a7aa47c48404465b94a36fd819fe3c7a P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260925 03:04:45.635639 24617 raft_consensus.cc:2454] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 6.
I20260925 03:04:45.637228 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9c33fd100ff44c1a3247319a324a044" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:45.638459 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "100c113a9fd84cc8adf2ec7b33ea0093" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:45.639078 26554 raft_consensus.cc:504] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:45.641913 26554 leader_election.cc:288] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:45.645704 26414 leader_election.cc:288] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:45.647715 24617 raft_consensus.cc:1226] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [term 6 LEADER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 5. Current term is 6. Ops: []
W20260925 03:04:45.648681 26533 log.cc:924] Time spent T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 1.110s user 0.003s sys 0.002s
I20260925 03:04:45.653146 24466 raft_consensus.cc:2454] T 8e5819c15eac4544a57e5ee6c9d4f117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 5.
I20260925 03:04:45.662533 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be4e0aa984f4f2b81a58e382e434ad5" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:45.664086 24618 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38570 (request call id 1388) took 1653 ms. Trace:
I20260925 03:04:45.664619 24618 rpcz_store.cc:276] 0925 03:04:44.010599 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:44.010863 (+ 264us) service_pool.cc:227] Handling call
0925 03:04:44.011248 (+ 385us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:04:44.060110 (+ 48862us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:44.060141 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:04:44.066189 (+ 6048us) log.cc:842] Serialized 26 byte log entry
0925 03:04:44.188192 (+122003us) raft_consensus.cc:1631] Marking committed up to 5
0925 03:04:44.188232 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:44.188308 (+ 76us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:04:45.640709 (+1452401us) spinlock_profiling.cc:243] Waited 10.6 ms on lock 0x7b44000232cc. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f43733ecf11 00007f43734baa95 00007f43734bb4b2 00007f4376163511 00007f437615ee1a 00007f4376152bee 00007f4376151f36 00007f4376151b58 00007f437615b551 00007f437615b509 00007f437615b499
0925 03:04:45.649622 (+ 8913us) raft_consensus.cc:1671] finished
0925 03:04:45.649646 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:45.664019 (+ 14373us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":68326,"raft.queue_time_us":49541,"raft.run_cpu_time_us":108,"raft.run_wall_time_us":3375,"spinlock_wait_cycles":24345984,"thread_start_us":105860,"threads_started":1,"wal-append.queue_time_us":213343}
I20260925 03:04:45.673574 24473 raft_consensus.cc:2454] T 25102ba08e4e49388f6176badcd10a25 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 3.
I20260925 03:04:45.671144 26497 consensus_queue.cc:1046] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260925 03:04:45.678555 26438 raft_consensus.cc:504] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:45.672143 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c81a1b3b616492ab6c964f45af36077" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:45.648499 26509 raft_consensus.cc:3037] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:45.687666 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c81a1b3b616492ab6c964f45af36077" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
W20260925 03:04:45.793378 24436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 85245.3us
W20260925 03:04:45.691093 26582 log.cc:924] Time spent T 7993f9c3e2a44f408bbf5d1e395e2306 P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.185s user 0.000s sys 0.006s
I20260925 03:04:45.682638 26262 leader_election.cc:288] T 420163a6cdce412ebc21c25fe3835638 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 6 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:45.685912 26414 raft_consensus.cc:482] T 04e068a0f80941dc9138cee063f53bab P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1bc1f2e62ab44af995165bd5116a9d43)
I20260925 03:04:45.686969 26591 raft_consensus.cc:482] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:45.792928 26584 raft_consensus.cc:482] T ceab6b0fccd54e909a64f66fe81a9a1f P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:45.723158 26545 raft_consensus.cc:482] T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:45.889639 26497 raft_consensus.cc:482] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:45.890312 26245 raft_consensus.cc:482] T b1e85248d3b04b979dd080caeeb02fdb P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:45.935578 26580 leader_election.cc:288] T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:45.972414 25194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c81a1b3b616492ab6c964f45af36077" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:46.003213 26386 raft_consensus.cc:482] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:45.862586 26507 raft_consensus.cc:482] T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:46.158295 25198 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48424 (request call id 208) took 1037 ms. Trace:
I20260925 03:04:46.162990 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c81a1b3b616492ab6c964f45af36077" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:46.164654 24473 raft_consensus.cc:1261] T 7993f9c3e2a44f408bbf5d1e395e2306 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260925 03:04:46.162919 25198 rpcz_store.cc:276] 0925 03:04:45.120443 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:45.122435 (+ 1992us) service_pool.cc:227] Handling call
0925 03:04:45.132235 (+ 9800us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:04:45.229908 (+ 97673us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:46.158024 (+928116us) spinlock_profiling.cc:243] Waited 97 ms on lock 0x7b54000ed154. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d0f51 00007f43764cf99e 00007f437b4112e9 00007f4374af7684 00007f4374af7612 00007f4374af7541 00007f4374af74bc 00007f4374af6772 00007f43748fa94c 00007f43748f9d86
0925 03:04:46.158231 (+ 207us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":223135104}
I20260925 03:04:46.157816 24548 raft_consensus.cc:1261] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Refusing update from remote peer 1bc1f2e62ab44af995165bd5116a9d43: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
W20260925 03:04:46.170369 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 796746us
W20260925 03:04:46.166286 26437 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.668s user 0.041s sys 0.086s
I20260925 03:04:46.172124 24548 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33068 (request call id 116) took 1503 ms. Trace:
I20260925 03:04:46.173413 24548 rpcz_store.cc:276] 0925 03:04:44.668827 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:45.366010 (+697183us) service_pool.cc:227] Handling call
0925 03:04:46.157177 (+791167us) spinlock_profiling.cc:243] Waited 4.57 ms on lock 0x7b5800003c98. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f43733ecf11 00007f4373570851 00007f4373570702 00007f437c460b09 00007f437b39c730 00007f437b46279b 00007f437b462821 00007f437b4262b0 00007f437b4117f9 00007f437b4112a2 00007f4374af7684
0925 03:04:46.157422 (+ 245us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:46.171832 (+ 14410us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:46.172068 (+ 236us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":10520448}
I20260925 03:04:46.004643 25162 leader_election.cc:302] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:46.185689 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "420163a6cdce412ebc21c25fe3835638" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:46.030180 26584 raft_consensus.cc:504] T ceab6b0fccd54e909a64f66fe81a9a1f P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:46.061188 26414 raft_consensus.cc:504] T 04e068a0f80941dc9138cee063f53bab P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:46.061136 26497 raft_consensus.cc:504] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:46.070930 26245 raft_consensus.cc:504] T b1e85248d3b04b979dd080caeeb02fdb P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:46.135710 26386 raft_consensus.cc:504] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:46.061874 26545 raft_consensus.cc:504] T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:46.109688 26591 raft_consensus.cc:504] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:46.190383 24548 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 871) took 1148 ms. Trace:
I20260925 03:04:46.201660 24621 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38570 (request call id 1395) took 1454 ms. Trace:
I20260925 03:04:46.291849 24548 rpcz_store.cc:276] 0925 03:04:45.042452 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:46.174456 (+1132004us) service_pool.cc:227] Handling call
0925 03:04:46.189892 (+ 15436us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:46.190060 (+ 168us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:46.190086 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:46.190122 (+ 36us) raft_consensus.cc:1631] Marking committed up to 5
0925 03:04:46.190150 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:46.190206 (+ 56us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:46.190328 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:46.293262 24621 rpcz_store.cc:276] 0925 03:04:44.747692 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:44.748103 (+ 411us) service_pool.cc:227] Handling call
0925 03:04:44.783491 (+ 35388us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:45.056956 (+273465us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:46.201336 (+1144380us) spinlock_profiling.cc:243] Waited 269 ms on lock 0x7b54000a5d54. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d0f51 00007f43764cf99e 00007f437b4112e9 00007f4374af7684 00007f4374af7612 00007f4374af7541 00007f4374af74bc 00007f4374af6772 00007f43748fa94c 00007f43748f9d86
0925 03:04:46.201588 (+ 252us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":619564928}
W20260925 03:04:46.171088 26437 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.673s user 0.041s sys 0.087s
I20260925 03:04:46.187701 26421 raft_consensus.cc:482] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260925 03:04:46.261996 26427 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.827s user 0.055s sys 0.120s
I20260925 03:04:46.301322 24548 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 872) took 1208 ms. Trace:
I20260925 03:04:46.305202 24617 raft_consensus.cc:1261] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260925 03:04:46.303682 24473 raft_consensus.cc:1261] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260925 03:04:46.305485 24548 rpcz_store.cc:276] 0925 03:04:45.093016 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:46.300184 (+1207168us) service_pool.cc:227] Handling call
0925 03:04:46.300602 (+ 418us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:46.300945 (+ 343us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:46.300976 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:46.301020 (+ 44us) raft_consensus.cc:1631] Marking committed up to 2
0925 03:04:46.301055 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:46.301128 (+ 73us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:46.301261 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:46.235800 26379 raft_consensus.cc:482] T 06f4a7eaf6534e4986c9fd35911038e4 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:46.189113 25196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1e85248d3b04b979dd080caeeb02fdb" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:46.157737 26507 raft_consensus.cc:504] T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:04:46.307605 26427 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.874s user 0.070s sys 0.128s
I20260925 03:04:46.225632 26209 consensus_queue.cc:1046] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.083s
I20260925 03:04:46.177858 26579 consensus_queue.cc:1035] T e6b6ec010caf4689b8c90449764f5cf9 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.130s
W20260925 03:04:46.310994 26594 log.cc:924] Time spent T edcf6e6a422d4d198d4fbb0f7d3d845d P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.597s user 0.000s sys 0.001s
I20260925 03:04:46.313243 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1e85248d3b04b979dd080caeeb02fdb" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:46.319329 24621 raft_consensus.cc:2379] T b1e85248d3b04b979dd080caeeb02fdb P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 in current term 5: Already voted for candidate eeab93abfdb141a895002ec9f312296c in this term.
I20260925 03:04:46.307298 26421 raft_consensus.cc:504] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:04:46.326118 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 144576us
W20260925 03:04:46.322767 26603 log.cc:924] Time spent T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5: Append to log took a long time: real 0.064s user 0.000s sys 0.001s
I20260925 03:04:46.322573 26591 leader_election.cc:288] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:46.322232 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:46.330258 24621 raft_consensus.cc:3037] T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:04:46.314520 26379 raft_consensus.cc:504] T 06f4a7eaf6534e4986c9fd35911038e4 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:46.331332 24547 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 837) took 2387 ms. Trace:
I20260925 03:04:46.332332 24547 rpcz_store.cc:276] 0925 03:04:43.944109 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:43.961337 (+ 17228us) service_pool.cc:227] Handling call
0925 03:04:43.978093 (+ 16756us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:04:44.186410 (+208317us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:44.186440 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:04:44.221292 (+ 34852us) log.cc:842] Serialized 26 byte log entry
0925 03:04:45.099372 (+878080us) raft_consensus.cc:1631] Marking committed up to 2
0925 03:04:45.099410 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:46.330905 (+1231495us) spinlock_profiling.cc:243] Waited 81.2 ms on lock 0x7b540036c254. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d0f51 00007f43764cf99e 00007f437b4112e9 00007f4374af7684 00007f4374af7612 00007f4374af7541 00007f4374af74bc 00007f4374af6772 00007f43748fa94c 00007f43748f9d86
0925 03:04:46.330972 (+ 67us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:04:46.331029 (+ 57us) raft_consensus.cc:1671] finished
0925 03:04:46.331044 (+ 15us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:46.331258 (+ 214us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":21249,"raft.run_cpu_time_us":98,"raft.run_wall_time_us":98,"spinlock_wait_cycles":186645760,"thread_start_us":988601,"threads_started":2,"wal-append.queue_time_us":932946,"wal-append.run_cpu_time_us":2780,"wal-append.run_wall_time_us":1066655}
I20260925 03:04:46.336352 26591 raft_consensus.cc:482] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:46.345223 26591 raft_consensus.cc:504] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:46.347577 26591 leader_election.cc:288] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:46.354305 26507 leader_election.cc:288] T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:46.355366 24621 raft_consensus.cc:2454] T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 6.
I20260925 03:04:46.358728 24615 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38570 (request call id 1396) took 1273 ms. Trace:
I20260925 03:04:46.359144 24615 rpcz_store.cc:276] 0925 03:04:45.085311 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:45.085498 (+ 187us) service_pool.cc:227] Handling call
0925 03:04:45.085833 (+ 335us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:04:45.400864 (+315031us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:46.358471 (+957607us) spinlock_profiling.cc:243] Waited 313 ms on lock 0x7b5400363654. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d0f51 00007f43764cf99e 00007f437b4112e9 00007f4374af7684 00007f4374af7612 00007f4374af7541 00007f4374af74bc 00007f4374af6772 00007f43748fa94c 00007f43748f9d86
0925 03:04:46.358664 (+ 193us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":719906560}
I20260925 03:04:46.363704 26421 leader_election.cc:288] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:46.382477 26558 raft_consensus.cc:482] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:46.383211 26558 raft_consensus.cc:504] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:46.383759 26401 raft_consensus.cc:3032] T 04e068a0f80941dc9138cee063f53bab P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 LEADER]: Stepping down as leader of term 2
I20260925 03:04:46.384162 26401 raft_consensus.cc:729] T 04e068a0f80941dc9138cee063f53bab P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1bc1f2e62ab44af995165bd5116a9d43, State: Running, Role: LEADER
I20260925 03:04:46.384904 26584 leader_election.cc:288] T ceab6b0fccd54e909a64f66fe81a9a1f P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:46.385968 26558 leader_election.cc:288] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:46.364101 26414 leader_election.cc:288] T 04e068a0f80941dc9138cee063f53bab P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:46.387782 26386 leader_election.cc:288] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:46.390515 24620 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38564 (request call id 1300) took 1365 ms. Trace:
I20260925 03:04:46.390856 24620 rpcz_store.cc:276] 0925 03:04:45.024615 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:45.025079 (+ 464us) service_pool.cc:227] Handling call
0925 03:04:46.390285 (+1365206us) spinlock_profiling.cc:243] Waited 56.6 ms on lock 0x7b54000a6254. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d42ae 00007f437b411c27 00007f4374af9c04 00007f4374af9b92 00007f4374af9ac1 00007f4374af9a3c 00007f4374af8cf2 00007f43748fa94c 00007f43748f9d86 00007f43748f972f
0925 03:04:46.390385 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":130251520}
W20260925 03:04:46.393180 26509 consensus_meta.cc:287] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c: Time spent flushing consensus metadata: real 0.694s user 0.010s sys 0.001s
I20260925 03:04:46.394075 26438 leader_election.cc:288] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:46.400005 26379 leader_election.cc:288] T 06f4a7eaf6534e4986c9fd35911038e4 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:46.403579 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35992316cb8d4c028333388eff02ecde" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:46.409089 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60d45b7d467d4e1ab92367f6d7d52b15" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
W20260925 03:04:46.413672 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 1.21471e+06us
I20260925 03:04:46.417753 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e5819c15eac4544a57e5ee6c9d4f117" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:46.416052 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5aec38e66a947c681e6c2a72b05e6ca" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 8 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:46.421656 24514 leader_election.cc:302] T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:46.421686 24548 raft_consensus.cc:2454] T 8e5819c15eac4544a57e5ee6c9d4f117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 3.
I20260925 03:04:46.423190 26545 leader_election.cc:288] T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:46.425614 24621 raft_consensus.cc:1226] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [term 5 LEADER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 4. Current term is 5. Ops: []
I20260925 03:04:46.425683 24543 raft_consensus.cc:1261] T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260925 03:04:45.100988 24542 rpcz_store.cc:276] 0925 03:04:44.077321 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:44.957764 (+880443us) service_pool.cc:227] Handling call
0925 03:04:44.958058 (+ 294us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:44.971840 (+ 13782us) raft_consensus.cc:1520] Early marking committed up to index 3
0925 03:04:44.982716 (+ 10876us) log.cc:842] Serialized 14 byte log entry
0925 03:04:45.092583 (+109867us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:45.092634 (+ 51us) raft_consensus.cc:1631] Marking committed up to 3
0925 03:04:45.092671 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:45.092751 (+ 80us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:45.092856 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":11457,"prepare.run_cpu_time_us":55,"prepare.run_wall_time_us":2477,"raft.queue_time_us":72,"raft.run_cpu_time_us":105,"raft.run_wall_time_us":4053,"thread_start_us":109493,"threads_started":1}
I20260925 03:04:46.430083 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be4e0aa984f4f2b81a58e382e434ad5" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:46.431054 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06f4a7eaf6534e4986c9fd35911038e4" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:46.432040 24617 raft_consensus.cc:2454] T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 5.
I20260925 03:04:46.423585 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25102ba08e4e49388f6176badcd10a25" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:46.433202 24548 raft_consensus.cc:3037] T 25102ba08e4e49388f6176badcd10a25 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:46.381034 26497 leader_election.cc:288] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:46.437942 26608 raft_consensus.cc:482] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:46.439620 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43366a0a2f9d4344bf4e37ffd290ef8a" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:46.440551 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceab6b0fccd54e909a64f66fe81a9a1f" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:46.413937 26412 raft_consensus.cc:2726] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260925 03:04:46.442535 24511 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.4:43979, user_credentials={real_user=slave}} blocked reactor thread for 969950us
I20260925 03:04:46.434465 26241 raft_consensus.cc:482] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:46.443480 26241 raft_consensus.cc:504] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:46.444082 26571 raft_consensus.cc:482] T cc2df40db08649c9a75e90000b878559 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:46.443439 26530 raft_consensus.cc:2726] T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260925 03:04:46.444109 24544 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 870) took 1402 ms. Trace:
I20260925 03:04:46.444536 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48ca6ccd98194cf2af7b5278d15bf4ea" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:46.445523 24544 rpcz_store.cc:276] 0925 03:04:45.042131 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:45.518635 (+476504us) service_pool.cc:227] Handling call
0925 03:04:45.520880 (+ 2245us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:45.521075 (+ 195us) raft_consensus.cc:1520] Early marking committed up to index 3
0925 03:04:45.524659 (+ 3584us) log.cc:842] Serialized 14 byte log entry
0925 03:04:46.443792 (+919133us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:46.443842 (+ 50us) raft_consensus.cc:1631] Marking committed up to 3
0925 03:04:46.443883 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:46.443956 (+ 73us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:46.444056 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":918853,"prepare.queue_time_us":3920,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":43,"raft.queue_time_us":918968,"raft.run_cpu_time_us":85,"raft.run_wall_time_us":86}
I20260925 03:04:46.447011 24547 raft_consensus.cc:2454] T 48ca6ccd98194cf2af7b5278d15bf4ea P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 3.
I20260925 03:04:46.445188 26412 raft_consensus.cc:2726] T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260925 03:04:46.473239 24615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f68804eb5a244d87bb395d50683c6012" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:46.473726 26357 raft_consensus.cc:482] T 7993f9c3e2a44f408bbf5d1e395e2306 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:46.474309 26357 raft_consensus.cc:504] T 7993f9c3e2a44f408bbf5d1e395e2306 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:46.444591 26571 raft_consensus.cc:504] T cc2df40db08649c9a75e90000b878559 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:46.478580 26571 leader_election.cc:288] T cc2df40db08649c9a75e90000b878559 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:46.479171 26571 raft_consensus.cc:482] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:46.479671 26571 raft_consensus.cc:504] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:46.474421 26497 consensus_queue.cc:1035] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20260925 03:04:46.438687 24546 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33068 (request call id 115) took 1789 ms (client timeout 1531 ms). Trace:
I20260925 03:04:46.490196 24542 raft_consensus.cc:2454] T 43366a0a2f9d4344bf4e37ffd290ef8a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 3.
W20260925 03:04:46.493184 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 71611.1us
I20260925 03:04:46.623761 24514 leader_election.cc:302] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:46.376865 26245 leader_election.cc:288] T b1e85248d3b04b979dd080caeeb02fdb P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
W20260925 03:04:46.622509 24546 rpcz_store.cc:269] 0925 03:04:44.635883 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:44.759707 (+123824us) service_pool.cc:227] Handling call
0925 03:04:46.425074 (+1665367us) spinlock_profiling.cc:243] Waited 49 ms on lock 0x7b54003b6854. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d42ae 00007f437b411c27 00007f4374af9c04 00007f4374af9b92 00007f4374af9ac1 00007f4374af9a3c 00007f4374af8cf2 00007f43748fa94c 00007f43748f9d86 00007f43748f972f
0925 03:04:46.425206 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":112673536}
I20260925 03:04:46.625749 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7691c6b475428c82d6181d693d26f3" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:46.633598 24548 raft_consensus.cc:2454] T 25102ba08e4e49388f6176badcd10a25 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 3.
I20260925 03:04:46.436825 26437 consensus_queue.cc:1035] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.295s
I20260925 03:04:46.440613 26438 consensus_queue.cc:1046] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.297s
I20260925 03:04:46.438548 26608 raft_consensus.cc:504] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:46.645594 26530 raft_consensus.cc:2726] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260925 03:04:46.442857 26529 raft_consensus.cc:482] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:47.341257 26529 raft_consensus.cc:504] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:46.809890 26357 leader_election.cc:288] T 7993f9c3e2a44f408bbf5d1e395e2306 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:47.344564 26438 consensus_queue.cc:1225] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Peer eeab93abfdb141a895002ec9f312296c log is divergent from this leader: its last log entry 5.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260925 03:04:46.808566 26571 leader_election.cc:288] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:47.352783 26241 leader_election.cc:288] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:47.353456 26529 leader_election.cc:288] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:47.360776 26522 raft_consensus.cc:3037] T b1e85248d3b04b979dd080caeeb02fdb P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:47.365847 26366 leader_election.cc:288] T 71f461a6e2ff434993b78a2d077f0abd P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:47.367635 25196 raft_consensus.cc:2454] T b1e85248d3b04b979dd080caeeb02fdb P 1bc1f2e62ab44af995165bd5116a9d43 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 5.
W20260925 03:04:47.377013 26546 log.cc:924] Time spent T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5: Append to log took a long time: real 2.545s user 0.009s sys 0.009s
I20260925 03:04:47.381778 26619 raft_consensus.cc:482] T 550118bc1a0641ce9803bdbf43f2069f P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:47.381956 26438 raft_consensus.cc:482] T edcf6e6a422d4d198d4fbb0f7d3d845d P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:47.382332 26619 raft_consensus.cc:504] T 550118bc1a0641ce9803bdbf43f2069f P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:47.382540 26438 raft_consensus.cc:504] T edcf6e6a422d4d198d4fbb0f7d3d845d P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:47.384459 26619 leader_election.cc:288] T 550118bc1a0641ce9803bdbf43f2069f P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:47.389468 26437 raft_consensus.cc:482] T 37a157996e944db2906110d87ecb0c54 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:47.390003 26437 raft_consensus.cc:504] T 37a157996e944db2906110d87ecb0c54 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:04:47.391680 26429 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.033s user 0.001s sys 0.004s
W20260925 03:04:47.392055 26429 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.034s user 0.001s sys 0.004s
I20260925 03:04:47.384596 26438 leader_election.cc:288] T edcf6e6a422d4d198d4fbb0f7d3d845d P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:47.392943 26438 raft_consensus.cc:482] T 7b8e0d260a884ecdb20f79de46c36b19 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:47.393513 26438 raft_consensus.cc:504] T 7b8e0d260a884ecdb20f79de46c36b19 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:47.395581 26438 leader_election.cc:288] T 7b8e0d260a884ecdb20f79de46c36b19 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:47.396103 26438 raft_consensus.cc:482] T 1414dcfe53f64a78ae413531eb8ec117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:47.392071 26437 leader_election.cc:288] T 37a157996e944db2906110d87ecb0c54 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:47.396536 26438 raft_consensus.cc:504] T 1414dcfe53f64a78ae413531eb8ec117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:47.399214 26438 leader_election.cc:288] T 1414dcfe53f64a78ae413531eb8ec117 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:47.405882 26371 raft_consensus.cc:482] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:47.406490 26371 raft_consensus.cc:504] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:47.408540 26371 leader_election.cc:288] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:47.409046 26371 raft_consensus.cc:482] T 3d89396576884f7c8901dba54f783947 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:47.409451 26371 raft_consensus.cc:504] T 3d89396576884f7c8901dba54f783947 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:47.411741 26371 leader_election.cc:288] T 3d89396576884f7c8901dba54f783947 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:47.412326 26371 raft_consensus.cc:482] T 100c113a9fd84cc8adf2ec7b33ea0093 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:47.412964 26371 raft_consensus.cc:504] T 100c113a9fd84cc8adf2ec7b33ea0093 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:47.415001 26371 leader_election.cc:288] T 100c113a9fd84cc8adf2ec7b33ea0093 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:47.418179 26437 raft_consensus.cc:482] T 293ef69caf514ac98510a68495f54eda P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1bc1f2e62ab44af995165bd5116a9d43)
I20260925 03:04:47.418391 26371 raft_consensus.cc:482] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:47.418956 26437 raft_consensus.cc:504] T 293ef69caf514ac98510a68495f54eda P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:47.419108 26371 raft_consensus.cc:504] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:47.417763 26619 raft_consensus.cc:482] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 1bc1f2e62ab44af995165bd5116a9d43)
I20260925 03:04:47.421800 26371 leader_election.cc:288] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:47.422289 26619 raft_consensus.cc:504] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:47.426493 26619 leader_election.cc:288] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:47.421949 26437 leader_election.cc:288] T 293ef69caf514ac98510a68495f54eda P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:47.440534 26555 raft_consensus.cc:3032] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 LEADER]: Stepping down as leader of term 5
I20260925 03:04:47.441043 26555 raft_consensus.cc:729] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:04:47.441912 26555 consensus_queue.cc:260] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:47.442925 26555 raft_consensus.cc:3037] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:04:47.451633 26554 raft_consensus.cc:482] T 29637f5a4a504ce6ba29033ccd0cd716 P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:47.453742 26337 raft_consensus.cc:482] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:47.454344 25199 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48408 (request call id 169) took 1766 ms. Trace:
W20260925 03:04:47.454869 25162 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 1.45034e+06us
I20260925 03:04:47.454736 26401 consensus_queue.cc:260] T 04e068a0f80941dc9138cee063f53bab P 1bc1f2e62ab44af995165bd5116a9d43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:47.455904 26592 raft_consensus.cc:2726] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260925 03:04:47.453792 26400 raft_consensus.cc:482] T 367277daf8d04778b3fa4ed94d71580d P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:47.457247 26400 raft_consensus.cc:504] T 367277daf8d04778b3fa4ed94d71580d P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:47.459825 26400 leader_election.cc:288] T 367277daf8d04778b3fa4ed94d71580d P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:47.454738 25199 rpcz_store.cc:276] 0925 03:04:45.687771 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:45.687912 (+ 141us) service_pool.cc:227] Handling call
0925 03:04:45.852524 (+164612us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:45.852814 (+ 290us) raft_consensus.cc:1520] Early marking committed up to index 7
0925 03:04:46.166851 (+314037us) log.cc:842] Serialized 14 byte log entry
0925 03:04:47.454002 (+1287151us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:47.454053 (+ 51us) raft_consensus.cc:1631] Marking committed up to 7
0925 03:04:47.454093 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:47.454158 (+ 65us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:47.454266 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1146053,"prepare.queue_time_us":383925,"prepare.run_cpu_time_us":91,"prepare.run_wall_time_us":3798,"raft.queue_time_us":1147284,"raft.run_cpu_time_us":90,"raft.run_wall_time_us":91,"thread_start_us":453130,"threads_started":2,"wal-append.queue_time_us":167304,"wal-append.run_cpu_time_us":1181,"wal-append.run_wall_time_us":1018181}
I20260925 03:04:47.452288 26554 raft_consensus.cc:504] T 29637f5a4a504ce6ba29033ccd0cd716 P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:47.454288 26337 raft_consensus.cc:504] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:47.464231 26337 leader_election.cc:288] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:47.465359 26554 leader_election.cc:288] T 29637f5a4a504ce6ba29033ccd0cd716 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:47.469136 26437 raft_consensus.cc:482] T a7aa47c48404465b94a36fd819fe3c7a P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:47.469702 26437 raft_consensus.cc:504] T a7aa47c48404465b94a36fd819fe3c7a P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:47.472005 26437 leader_election.cc:288] T a7aa47c48404465b94a36fd819fe3c7a P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:47.495301 25199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43"
I20260925 03:04:47.496068 25199 raft_consensus.cc:3037] T 7e49248643df4131a1c42e74244e05c2 P 1bc1f2e62ab44af995165bd5116a9d43 [term 5 FOLLOWER]: Advancing to term 6
W20260925 03:04:47.497113 24539 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.748s user 0.004s sys 0.008s
W20260925 03:04:47.497464 24539 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.749s user 0.004s sys 0.008s
I20260925 03:04:47.498178 24539 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 1362) took 2637 ms. Trace:
I20260925 03:04:47.498967 25194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e068a0f80941dc9138cee063f53bab" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:47.498528 24539 rpcz_store.cc:276] 0925 03:04:44.860678 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:45.439804 (+579126us) service_pool.cc:227] Handling call
0925 03:04:45.453022 (+ 13218us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:45.453255 (+ 233us) raft_consensus.cc:1520] Early marking committed up to index 4
0925 03:04:45.453572 (+ 317us) log.cc:842] Serialized 14 byte log entry
0925 03:04:46.641158 (+1187586us) log.cc:842] Serialized 14 byte log entry
0925 03:04:47.497840 (+856682us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:47.497897 (+ 57us) raft_consensus.cc:1631] Marking committed up to 4
0925 03:04:47.497936 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:47.498007 (+ 71us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:47.498127 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1072037,"prepare.queue_time_us":916275,"prepare.run_cpu_time_us":81,"prepare.run_wall_time_us":84,"raft.queue_time_us":991255,"raft.run_cpu_time_us":69,"raft.run_wall_time_us":69,"thread_start_us":947911,"threads_started":2,"wal-append.queue_time_us":717990}
I20260925 03:04:47.501114 25190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5aec38e66a947c681e6c2a72b05e6ca" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 8 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:47.501905 25190 raft_consensus.cc:2454] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 7.
I20260925 03:04:47.502444 24539 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 878) took 1084 ms. Trace:
I20260925 03:04:47.502797 26529 raft_consensus.cc:482] T 442d4a29b1d743599f4b6b4871e26668 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:47.503616 24473 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1030) took 1336 ms. Trace:
I20260925 03:04:47.503417 26529 raft_consensus.cc:504] T 442d4a29b1d743599f4b6b4871e26668 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:47.504004 25198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 7 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:47.504949 25198 raft_consensus.cc:2454] T 29637f5a4a504ce6ba29033ccd0cd716 P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 6.
I20260925 03:04:47.504580 25190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1e85248d3b04b979dd080caeeb02fdb" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:47.504004 24539 rpcz_store.cc:276] 0925 03:04:46.418211 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:47.501428 (+1083217us) service_pool.cc:227] Handling call
0925 03:04:47.501848 (+ 420us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:47.502027 (+ 179us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:47.502055 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:47.502095 (+ 40us) raft_consensus.cc:1631] Marking committed up to 2
0925 03:04:47.502128 (+ 33us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:47.502183 (+ 55us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:47.502370 (+ 187us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:47.507766 24539 raft_consensus.cc:1261] T a7aa47c48404465b94a36fd819fe3c7a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260925 03:04:47.508320 24539 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 880) took 1089 ms. Trace:
I20260925 03:04:47.508601 25199 raft_consensus.cc:2454] T 7e49248643df4131a1c42e74244e05c2 P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 6.
I20260925 03:04:47.503993 24473 rpcz_store.cc:276] 0925 03:04:46.167459 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:46.167645 (+ 186us) service_pool.cc:227] Handling call
0925 03:04:46.188569 (+ 20924us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:46.306259 (+117690us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:47.503408 (+1197149us) spinlock_profiling.cc:243] Waited 115 ms on lock 0x7b54002c0954. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d0f51 00007f43764cf99e 00007f437b4112e9 00007f4374af7684 00007f4374af7612 00007f4374af7541 00007f4374af74bc 00007f4374af6772 00007f43748fa94c 00007f43748f9d86
0925 03:04:47.503564 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":263885184}
I20260925 03:04:47.511655 24546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 881) took 1092 ms. Trace:
I20260925 03:04:47.511973 24546 rpcz_store.cc:276] 0925 03:04:46.418946 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:47.511052 (+1092106us) service_pool.cc:227] Handling call
0925 03:04:47.511344 (+ 292us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:47.511471 (+ 127us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:47.511485 (+ 14us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:47.511509 (+ 24us) raft_consensus.cc:1631] Marking committed up to 4
0925 03:04:47.511526 (+ 17us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:47.511554 (+ 28us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:47.511617 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:47.513211 24546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 882) took 1094 ms. Trace:
I20260925 03:04:47.513669 24539 rpcz_store.cc:276] 0925 03:04:46.418686 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:47.507090 (+1088404us) service_pool.cc:227] Handling call
0925 03:04:47.507427 (+ 337us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:47.508086 (+ 659us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:47.508270 (+ 184us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:47.513437 24546 rpcz_store.cc:276] 0925 03:04:46.419183 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:47.512606 (+1093423us) service_pool.cc:227] Handling call
0925 03:04:47.512906 (+ 300us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:47.513024 (+ 118us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:47.513037 (+ 13us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:47.513060 (+ 23us) raft_consensus.cc:1631] Marking committed up to 3
0925 03:04:47.513076 (+ 16us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:47.513107 (+ 31us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:47.513172 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:47.515734 24539 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 884) took 1092 ms. Trace:
I20260925 03:04:47.516119 24539 rpcz_store.cc:276] 0925 03:04:46.423197 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:47.514903 (+1091706us) service_pool.cc:227] Handling call
0925 03:04:47.515225 (+ 322us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:47.515399 (+ 174us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:47.515437 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:47.515486 (+ 49us) raft_consensus.cc:1631] Marking committed up to 2
0925 03:04:47.515519 (+ 33us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:47.515575 (+ 56us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:47.515680 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:47.518512 24539 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 1367) took 1072 ms. Trace:
I20260925 03:04:47.518890 24539 rpcz_store.cc:276] 0925 03:04:46.445944 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:47.517496 (+1071552us) service_pool.cc:227] Handling call
0925 03:04:47.517853 (+ 357us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:47.518175 (+ 322us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:47.518206 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:47.518251 (+ 45us) raft_consensus.cc:1631] Marking committed up to 3
0925 03:04:47.518281 (+ 30us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:47.518337 (+ 56us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:47.518460 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":98,"raft.run_cpu_time_us":84,"raft.run_wall_time_us":84}
I20260925 03:04:47.519717 24546 raft_consensus.cc:1261] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260925 03:04:47.521346 24546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 1365) took 1077 ms. Trace:
I20260925 03:04:47.521718 24546 rpcz_store.cc:276] 0925 03:04:46.444246 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:47.519074 (+1074828us) service_pool.cc:227] Handling call
0925 03:04:47.519386 (+ 312us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:47.521119 (+ 1733us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:47.521297 (+ 178us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:47.529208 26598 raft_consensus.cc:3032] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 LEADER]: Stepping down as leader of term 4
I20260925 03:04:47.529675 26598 raft_consensus.cc:729] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:04:47.530480 26598 consensus_queue.cc:260] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:47.531664 26598 raft_consensus.cc:3037] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:47.531006 24546 raft_consensus.cc:1261] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260925 03:04:47.533874 26555 raft_consensus.cc:482] T b4ebd2b043024a548005c8b6b00b0449 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:47.534436 26555 raft_consensus.cc:504] T b4ebd2b043024a548005c8b6b00b0449 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:47.534742 26577 raft_consensus.cc:482] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 1bc1f2e62ab44af995165bd5116a9d43)
I20260925 03:04:47.537155 26577 raft_consensus.cc:504] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:47.536547 26555 leader_election.cc:288] T b4ebd2b043024a548005c8b6b00b0449 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:47.538928 26555 raft_consensus.cc:482] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:47.539422 26555 raft_consensus.cc:504] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:47.541457 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b627ecb4a5384ef392aa695e06599c39" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:47.542186 24546 raft_consensus.cc:2454] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 5.
I20260925 03:04:47.544785 24622 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34546 (request call id 123) took 2194 ms. Trace:
I20260925 03:04:47.545804 25196 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48408 (request call id 170) took 1432 ms. Trace:
I20260925 03:04:47.545941 26241 raft_consensus.cc:482] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:47.546772 26241 raft_consensus.cc:504] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:47.546231 25196 rpcz_store.cc:276] 0925 03:04:46.113037 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:46.119088 (+ 6051us) service_pool.cc:227] Handling call
0925 03:04:47.545611 (+1426523us) spinlock_profiling.cc:243] Waited 1.05 s on lock 0x7b54004c0254. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d42ae 00007f437b411c27 00007f4374af9c04 00007f4374af9b92 00007f4374af9ac1 00007f4374af9a3c 00007f4374af8cf2 00007f43748fa94c 00007f43748f9d86 00007f43748f972f
0925 03:04:47.545723 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2425227520}
I20260925 03:04:47.551146 26555 leader_election.cc:288] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
W20260925 03:04:47.548650 24617 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.796s user 0.010s sys 0.021s
W20260925 03:04:47.551851 24617 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.799s user 0.012s sys 0.021s
I20260925 03:04:47.552357 24543 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 848) took 3537 ms. Trace:
I20260925 03:04:47.552497 24617 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38570 (request call id 1422) took 1131 ms. Trace:
I20260925 03:04:47.554701 24618 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38570 (request call id 1404) took 1869 ms. Trace:
I20260925 03:04:47.557093 25193 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48408 (request call id 168) took 1925 ms. Trace:
I20260925 03:04:47.558342 24618 rpcz_store.cc:276] 0925 03:04:45.685569 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:45.685710 (+ 141us) service_pool.cc:227] Handling call
0925 03:04:46.151721 (+466011us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:46.310379 (+158658us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:46.310407 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:46.310451 (+ 44us) raft_consensus.cc:1631] Marking committed up to 4
0925 03:04:46.310478 (+ 27us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:47.554484 (+1244006us) spinlock_profiling.cc:243] Waited 158 ms on lock 0x7b54000735d4. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d0f51 00007f43764cf99e 00007f437b4112e9 00007f4374af7684 00007f4374af7612 00007f4374af7541 00007f4374af74bc 00007f4374af6772 00007f43748fa94c 00007f43748f9d86
0925 03:04:47.554526 (+ 42us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:47.554647 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":364369280}
I20260925 03:04:47.558079 24543 rpcz_store.cc:276] 0925 03:04:44.014839 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:44.927571 (+912732us) service_pool.cc:227] Handling call
0925 03:04:46.425317 (+1497746us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:46.426183 (+ 866us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:47.552163 (+1125980us) spinlock_profiling.cc:243] Waited 1.5 s on lock 0x7b54003b6850. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764cfa09 00007f437b4112e9 00007f4374af7684 00007f4374af7612 00007f4374af7541 00007f4374af74bc 00007f4374af6772 00007f43748fa94c 00007f43748f9d86 00007f43748f972f
0925 03:04:47.552299 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3443648896}
I20260925 03:04:47.558234 24617 rpcz_store.cc:276] 0925 03:04:46.420934 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:46.451644 (+ 30710us) service_pool.cc:227] Handling call
0925 03:04:46.492275 (+ 40631us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:04:46.492504 (+ 229us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:46.492529 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:04:46.492982 (+ 453us) log.cc:842] Serialized 26 byte log entry
0925 03:04:47.552175 (+1059193us) raft_consensus.cc:1631] Marking committed up to 3
0925 03:04:47.552214 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:47.552282 (+ 68us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:04:47.552334 (+ 52us) raft_consensus.cc:1671] finished
0925 03:04:47.552344 (+ 10us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:47.552449 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":799067,"threads_started":1,"wal-append.queue_time_us":638712}
I20260925 03:04:47.559679 25193 rpcz_store.cc:276] 0925 03:04:45.631337 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:45.632807 (+ 1470us) service_pool.cc:227] Handling call
0925 03:04:45.633117 (+ 310us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:45.643939 (+ 10822us) raft_consensus.cc:1520] Early marking committed up to index 6
0925 03:04:46.004185 (+360246us) log.cc:842] Serialized 14 byte log entry
0925 03:04:47.455614 (+1451429us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:47.455671 (+ 57us) raft_consensus.cc:1631] Marking committed up to 6
0925 03:04:47.455717 (+ 46us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:47.556845 (+101128us) spinlock_profiling.cc:243] Waited 10.5 ms on lock 0x7b54004bcdd4. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d0f51 00007f43764cf99e 00007f437b4112e9 00007f4374af7684 00007f4374af7612 00007f4374af7541 00007f4374af74bc 00007f4374af6772 00007f43748fa94c 00007f43748f9d86
0925 03:04:47.556900 (+ 55us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:47.557033 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1164163,"prepare.queue_time_us":548201,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":46,"raft.queue_time_us":1164829,"raft.run_cpu_time_us":75,"raft.run_wall_time_us":75,"spinlock_wait_cycles":24244096,"thread_start_us":520603,"threads_started":2,"wal-append.queue_time_us":307936,"wal-append.run_cpu_time_us":1202,"wal-append.run_wall_time_us":1017547}
I20260925 03:04:47.548908 26241 leader_election.cc:288] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:47.545188 24622 rpcz_store.cc:276] 0925 03:04:45.350355 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:45.350506 (+ 151us) service_pool.cc:227] Handling call
0925 03:04:45.369945 (+ 19439us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:04:45.387962 (+ 18017us) raft_consensus.cc:1520] Early marking committed up to index 7
0925 03:04:46.369554 (+981592us) log.cc:842] Serialized 14 byte log entry
0925 03:04:46.791378 (+421824us) log.cc:842] Serialized 14 byte log entry
0925 03:04:46.791663 (+ 285us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:04:47.413655 (+621992us) spinlock_profiling.cc:243] Waited 6.19 ms on lock 0x7b180032fa40. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f437c4478b1 00007f437c454661 00007f43735a8638 00007f43735abb11 00007f43735abac9 00007f43735aba59 00007f43735aba21 00007f43735aad1d 00007f437c80c204 00007f437c80c039 00007f437358adc6
0925 03:04:47.534400 (+120745us) log.cc:842] Serialized 26 byte log entry
0925 03:04:47.534562 (+ 162us) raft_consensus.cc:1631] Marking committed up to 7
0925 03:04:47.534596 (+ 34us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:47.534674 (+ 78us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:04:47.544519 (+ 9845us) raft_consensus.cc:1671] finished
0925 03:04:47.544552 (+ 33us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:47.544718 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":976468,"prepare.queue_time_us":977322,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":33,"raft.queue_time_us":8167,"raft.run_cpu_time_us":491,"raft.run_wall_time_us":18281,"spinlock_wait_cycles":34352768,"thread_start_us":420983,"threads_started":1,"wal-append.queue_time_us":981340}
I20260925 03:04:47.564232 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06f4a7eaf6534e4986c9fd35911038e4" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:47.565501 24539 raft_consensus.cc:1261] T 7993f9c3e2a44f408bbf5d1e395e2306 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260925 03:04:47.565739 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 7 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:47.566776 24543 raft_consensus.cc:2454] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 6.
I20260925 03:04:47.567487 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e068a0f80941dc9138cee063f53bab" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:47.568884 24546 raft_consensus.cc:2379] T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c in current term 3: Already voted for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in this term.
I20260925 03:04:47.589002 26508 raft_consensus.cc:3037] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:47.593065 26293 raft_consensus.cc:3037] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:47.596071 26401 raft_consensus.cc:3037] T 04e068a0f80941dc9138cee063f53bab P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:47.598420 26270 raft_consensus.cc:482] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:47.599011 26270 raft_consensus.cc:504] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:47.600672 26607 raft_consensus.cc:482] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:47.601223 26607 raft_consensus.cc:504] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:47.601742 24436 leader_election.cc:302] T 06f4a7eaf6534e4986c9fd35911038e4 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:47.602514 26598 raft_consensus.cc:2781] T 06f4a7eaf6534e4986c9fd35911038e4 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260925 03:04:47.602962 26598 raft_consensus.cc:482] T 06f4a7eaf6534e4986c9fd35911038e4 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:47.603368 26598 raft_consensus.cc:3037] T 06f4a7eaf6534e4986c9fd35911038e4 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:04:47.605769 25194 raft_consensus.cc:2454] T 04e068a0f80941dc9138cee063f53bab P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 3.
I20260925 03:04:47.606216 25193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "293ef69caf514ac98510a68495f54eda" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:47.606607 25196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e54ca1757ee4e42956a9cebbeca4f62" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:47.607163 26607 leader_election.cc:288] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:47.608137 26619 raft_consensus.cc:482] T c85f5d1d3f004cf5817bb594fd8e6469 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
W20260925 03:04:47.609107 25163 leader_election.cc:334] T a4abe0ea757e4886be4945c5001e529a P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389): Timed out: RequestConsensusVote RPC to 127.22.193.2:36389 timed out after 1.516s (SENT)
I20260925 03:04:47.609467 26619 raft_consensus.cc:504] T c85f5d1d3f004cf5817bb594fd8e6469 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:47.609725 25163 leader_election.cc:302] T a4abe0ea757e4886be4945c5001e529a P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:47.610502 26401 raft_consensus.cc:2726] T a4abe0ea757e4886be4945c5001e529a P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260925 03:04:47.611119 26621 raft_consensus.cc:482] T 4e5332a5d329493991dff994e70ea92c P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:47.611328 26619 leader_election.cc:288] T c85f5d1d3f004cf5817bb594fd8e6469 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
W20260925 03:04:47.611928 25163 leader_election.cc:334] T e92f377a79354a6ebb4eec67303c13e3 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389): Timed out: RequestConsensusVote RPC to 127.22.193.2:36389 timed out after 1.531s (SENT)
I20260925 03:04:47.611614 26621 raft_consensus.cc:504] T 4e5332a5d329493991dff994e70ea92c P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:47.612380 25163 leader_election.cc:302] T e92f377a79354a6ebb4eec67303c13e3 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:47.613591 26401 raft_consensus.cc:2726] T e92f377a79354a6ebb4eec67303c13e3 P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260925 03:04:47.614437 26621 leader_election.cc:288] T 4e5332a5d329493991dff994e70ea92c P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:47.610864 26371 raft_consensus.cc:482] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:47.611279 24616 raft_consensus.cc:2454] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 4.
I20260925 03:04:47.615636 26371 raft_consensus.cc:504] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:47.617249 26598 raft_consensus.cc:504] T 06f4a7eaf6534e4986c9fd35911038e4 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:04:47.618046 24616 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38564 (request call id 1301) took 2285 ms (client timeout 1714 ms). Trace:
W20260925 03:04:47.618482 24616 rpcz_store.cc:269] 0925 03:04:45.332233 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:45.332835 (+ 602us) service_pool.cc:227] Handling call
0925 03:04:47.617849 (+2285014us) spinlock_profiling.cc:243] Waited 2.26 s on lock 0x7b540018f1d4. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d42ae 00007f437b411c27 00007f4374af9c04 00007f4374af9b92 00007f4374af9ac1 00007f4374af9a3c 00007f4374af8cf2 00007f43748fa94c 00007f43748f9d86 00007f43748f972f
0925 03:04:47.617963 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5207839104}
I20260925 03:04:47.619341 26598 leader_election.cc:288] T 06f4a7eaf6534e4986c9fd35911038e4 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:47.623224 24541 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 1358) took 3369 ms. Trace:
I20260925 03:04:47.628208 25191 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48408 (request call id 167) took 2248 ms. Trace:
I20260925 03:04:47.628669 25191 rpcz_store.cc:276] 0925 03:04:45.380072 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:45.382058 (+ 1986us) service_pool.cc:227] Handling call
0925 03:04:47.535656 (+2153598us) spinlock_profiling.cc:243] Waited 1.89 ms on lock 0x7b5800004e98. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f43733ecf11 00007f4373570851 00007f4373570702 00007f437c460b09 00007f437b39c730 00007f437b46279b 00007f437b462821 00007f437b4262b0 00007f437b4117f9 00007f437b4112a2 00007f4374af7684
0925 03:04:47.535888 (+ 232us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:04:47.536266 (+ 378us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:47.536295 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:04:47.536805 (+ 510us) log.cc:842] Serialized 26 byte log entry
0925 03:04:47.578672 (+ 41867us) raft_consensus.cc:1631] Marking committed up to 4
0925 03:04:47.578721 (+ 49us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:47.578831 (+ 110us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:04:47.627817 (+ 48986us) raft_consensus.cc:1671] finished
0925 03:04:47.627844 (+ 27us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:47.628120 (+ 276us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":4218,"raft.run_cpu_time_us":124,"raft.run_wall_time_us":124,"spinlock_wait_cycles":4356352,"thread_start_us":41447,"threads_started":1,"wal-append.queue_time_us":80054}
I20260925 03:04:47.623605 24541 rpcz_store.cc:276] 0925 03:04:44.253768 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:45.273601 (+1019833us) service_pool.cc:227] Handling call
0925 03:04:45.273889 (+ 288us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:04:46.436259 (+1162370us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:46.436290 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:04:46.442589 (+ 6299us) log.cc:842] Serialized 26 byte log entry
0925 03:04:46.720174 (+277585us) raft_consensus.cc:1631] Marking committed up to 1
0925 03:04:46.720220 (+ 46us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:46.720299 (+ 79us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:04:47.622996 (+902697us) raft_consensus.cc:1671] finished
0925 03:04:47.623017 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:47.623163 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1145269,"raft.queue_time_us":1155074,"raft.run_cpu_time_us":204,"raft.run_wall_time_us":8285,"thread_start_us":277199,"threads_started":1,"wal-append.queue_time_us":918430}
I20260925 03:04:47.635674 26598 raft_consensus.cc:482] T f22ddd7d1c444dc58f45a1fb0b3e02bc P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:47.636332 26598 raft_consensus.cc:504] T f22ddd7d1c444dc58f45a1fb0b3e02bc P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:47.638911 26598 leader_election.cc:288] T f22ddd7d1c444dc58f45a1fb0b3e02bc P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:47.644339 26608 leader_election.cc:288] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:47.644940 26438 raft_consensus.cc:482] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:47.645529 26438 raft_consensus.cc:504] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:47.647646 26438 leader_election.cc:288] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:47.648376 24616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bb46bce3564841b7674e6a693bd2e6" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:47.644944 26608 raft_consensus.cc:482] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:47.617832 26371 leader_election.cc:288] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:47.650337 26608 raft_consensus.cc:504] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:47.651336 24436 leader_election.cc:302] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:04:47.652307 26608 leader_election.cc:288] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:47.652856 26438 raft_consensus.cc:2781] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260925 03:04:47.654588 26438 raft_consensus.cc:482] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:47.655017 26438 raft_consensus.cc:3037] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Advancing to term 8
I20260925 03:04:46.757499 24613 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38564 (request call id 1298) took 2428 ms. Trace:
I20260925 03:04:47.656746 24613 rpcz_store.cc:276] 0925 03:04:44.301915 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:44.305387 (+ 3472us) service_pool.cc:227] Handling call
0925 03:04:44.438229 (+132842us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:04:44.723298 (+285069us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:44.757828 (+ 34530us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:04:44.830803 (+ 72975us) log.cc:842] Serialized 26 byte log entry
0925 03:04:45.024901 (+194098us) raft_consensus.cc:1631] Marking committed up to 6
0925 03:04:46.357975 (+1333074us) log.cc:842] Serialized 14 byte log entry
0925 03:04:46.437783 (+ 79808us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:46.438887 (+ 1104us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:04:46.622430 (+183543us) raft_consensus.cc:1671] finished
0925 03:04:46.622459 (+ 29us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:46.730659 (+108200us) inbound_call.cc:173] Queueing success response
0925 03:04:47.397608 (+666949us) spinlock_profiling.cc:243] Waited 3.88 ms on lock 0x7b48000c0258. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f437b3777f1 00007f437b36e8e7 00007f437b36e77a 00007f437b46295f 00007f437b462627 00007f437b476b2c 00007f437b476ad8 00007f437b476a58 00007f437b476a13 00007f437b475cbc 00007f437a3f7be6
Metrics: {"mutex_wait_us":1459500,"prepare.queue_time_us":1331013,"prepare.run_cpu_time_us":27,"prepare.run_wall_time_us":28,"raft.queue_time_us":119410,"raft.run_cpu_time_us":1116,"raft.run_wall_time_us":1010566,"spinlock_wait_cycles":8921344,"thread_start_us":212562,"threads_started":2,"wal-append.queue_time_us":641434,"wal-append.run_cpu_time_us":3155,"wal-append.run_wall_time_us":1126841}
I20260925 03:04:47.660189 24588 leader_election.cc:302] T 48ca6ccd98194cf2af7b5278d15bf4ea P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:47.661163 26607 raft_consensus.cc:2781] T 48ca6ccd98194cf2af7b5278d15bf4ea P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260925 03:04:47.661562 26607 raft_consensus.cc:482] T 48ca6ccd98194cf2af7b5278d15bf4ea P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:47.661926 26607 raft_consensus.cc:3037] T 48ca6ccd98194cf2af7b5278d15bf4ea P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:47.661404 26448 consensus_queue.cc:1035] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260925 03:04:47.652585 24436 leader_election.cc:302] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:47.670495 26621 raft_consensus.cc:2781] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader election won for term 6
I20260925 03:04:47.671032 26621 raft_consensus.cc:686] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 LEADER]: Becoming Leader. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:04:47.671954 26621 consensus_queue.cc:237] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 6, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:47.673630 24614 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38564 (request call id 1299) took 2649 ms. Trace:
I20260925 03:04:47.674021 24614 rpcz_store.cc:276] 0925 03:04:45.024294 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:45.029683 (+ 5389us) service_pool.cc:227] Handling call
0925 03:04:45.030040 (+ 357us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:04:45.097742 (+ 67702us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:45.097777 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:04:45.121364 (+ 23587us) log.cc:842] Serialized 26 byte log entry
0925 03:04:45.190164 (+ 68800us) raft_consensus.cc:1631] Marking committed up to 5
0925 03:04:45.190206 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:47.673205 (+2482999us) spinlock_profiling.cc:243] Waited 53 ms on lock 0x7b5400217ad4. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d0f51 00007f43764cf99e 00007f437b4112e9 00007f4374af7684 00007f4374af7612 00007f4374af7541 00007f4374af74bc 00007f4374af6772 00007f43748fa94c 00007f43748f9d86
0925 03:04:47.673254 (+ 49us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:04:47.673300 (+ 46us) raft_consensus.cc:1671] finished
0925 03:04:47.673311 (+ 11us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:47.673563 (+ 252us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":21512,"raft.queue_time_us":9566,"raft.run_cpu_time_us":178,"raft.run_wall_time_us":18804,"spinlock_wait_cycles":122010496,"thread_start_us":34624,"threads_started":1,"wal-append.queue_time_us":108104,"wal-append.run_cpu_time_us":1877,"wal-append.run_wall_time_us":1105151}
I20260925 03:04:47.677131 24436 leader_election.cc:302] T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:04:47.680297 26270 leader_election.cc:288] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:47.681056 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367277daf8d04778b3fa4ed94d71580d" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:47.689716 24616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "550118bc1a0641ce9803bdbf43f2069f" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:47.690285 24464 raft_consensus.cc:2454] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 4.
I20260925 03:04:47.681668 26619 raft_consensus.cc:2781] T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260925 03:04:47.691177 26619 raft_consensus.cc:482] T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 1bc1f2e62ab44af995165bd5116a9d43)
I20260925 03:04:47.691571 26619 raft_consensus.cc:3037] T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:47.695399 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60d45b7d467d4e1ab92367f6d7d52b15" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:47.696153 24546 raft_consensus.cc:2454] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 6.
I20260925 03:04:47.682585 24468 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 975) took 3536 ms. Trace:
I20260925 03:04:47.683444 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 7 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:47.684546 26499 raft_consensus.cc:482] T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:47.699213 26499 raft_consensus.cc:504] T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:47.701042 26621 raft_consensus.cc:482] T 11bb46bce3564841b7674e6a693bd2e6 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:47.701434 26499 leader_election.cc:288] T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:47.689224 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcf6e6a422d4d198d4fbb0f7d3d845d" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:47.701591 26621 raft_consensus.cc:504] T 11bb46bce3564841b7674e6a693bd2e6 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:47.697329 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7691c6b475428c82d6181d693d26f3" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:47.703141 24539 raft_consensus.cc:2454] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 6.
I20260925 03:04:47.698176 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceab6b0fccd54e909a64f66fe81a9a1f" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:47.704241 26621 leader_election.cc:288] T 11bb46bce3564841b7674e6a693bd2e6 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:47.704713 24543 raft_consensus.cc:2379] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 in current term 5: Already voted for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in this term.
I20260925 03:04:47.705183 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:47.705544 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b8e0d260a884ecdb20f79de46c36b19" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:47.706207 24543 raft_consensus.cc:2454] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 4.
I20260925 03:04:47.706640 26621 raft_consensus.cc:482] T cc2df40db08649c9a75e90000b878559 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:47.707168 26621 raft_consensus.cc:504] T cc2df40db08649c9a75e90000b878559 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:47.696928 24468 rpcz_store.cc:276] 0925 03:04:44.145847 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:44.145987 (+ 140us) service_pool.cc:227] Handling call
0925 03:04:44.168059 (+ 22072us) raft_consensus.cc:1457] Updating replica for 2 ops
0925 03:04:44.189905 (+ 21846us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:44.189941 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0925 03:04:44.190506 (+ 565us) log.cc:842] Serialized 52 byte log entry
0925 03:04:44.638222 (+447716us) raft_consensus.cc:1631] Marking committed up to 2
0925 03:04:44.639469 (+ 1247us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:44.639557 (+ 88us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:04:47.376434 (+2736877us) spinlock_profiling.cc:243] Waited 2.66 ms on lock 0x7b44000232cc. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f43733ecf11 00007f43734baa95 00007f43734bb4b2 00007f4376163511 00007f437615ee1a 00007f4376152bee 00007f4376151f36 00007f4376151b58 00007f437615b551 00007f437615b509 00007f437615b499
0925 03:04:47.682253 (+305819us) raft_consensus.cc:1671] finished
0925 03:04:47.682276 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:47.682504 (+ 228us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":11456,"raft.run_cpu_time_us":143,"raft.run_wall_time_us":11114,"spinlock_wait_cycles":6111488,"thread_start_us":360582,"threads_started":1,"wal-append.queue_time_us":572851}
I20260925 03:04:47.708634 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "420163a6cdce412ebc21c25fe3835638" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:47.709192 26621 leader_election.cc:288] T cc2df40db08649c9a75e90000b878559 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:47.707255 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "550118bc1a0641ce9803bdbf43f2069f" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:47.710068 24543 raft_consensus.cc:2454] T 550118bc1a0641ce9803bdbf43f2069f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 4.
I20260925 03:04:47.710139 26621 raft_consensus.cc:482] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:47.710714 26621 raft_consensus.cc:504] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:47.704181 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35992316cb8d4c028333388eff02ecde" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:47.712816 26621 leader_election.cc:288] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:47.710968 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcf6e6a422d4d198d4fbb0f7d3d845d" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:47.712594 24539 raft_consensus.cc:2396] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 for term 7 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260925 03:04:47.713418 24543 raft_consensus.cc:2454] T edcf6e6a422d4d198d4fbb0f7d3d845d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 3.
I20260925 03:04:47.714084 24547 raft_consensus.cc:3037] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:04:47.714311 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1414dcfe53f64a78ae413531eb8ec117" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:47.714906 24543 raft_consensus.cc:2454] T 1414dcfe53f64a78ae413531eb8ec117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 2.
I20260925 03:04:47.715682 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37a157996e944db2906110d87ecb0c54" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:47.716217 24543 raft_consensus.cc:2454] T 37a157996e944db2906110d87ecb0c54 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 5.
I20260925 03:04:47.716992 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06f4a7eaf6534e4986c9fd35911038e4" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:47.717115 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f68804eb5a244d87bb395d50683c6012" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:47.717662 24546 raft_consensus.cc:2454] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 3.
I20260925 03:04:47.709270 24544 raft_consensus.cc:3037] T 420163a6cdce412ebc21c25fe3835638 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:04:47.720646 26630 raft_consensus.cc:482] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:47.721235 26630 raft_consensus.cc:504] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:47.724660 26630 leader_election.cc:288] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:47.726917 24465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06f4a7eaf6534e4986c9fd35911038e4" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:47.729555 24465 raft_consensus.cc:2379] T 06f4a7eaf6534e4986c9fd35911038e4 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c in current term 6: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:04:47.730448 24465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceab6b0fccd54e909a64f66fe81a9a1f" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:47.731070 24465 raft_consensus.cc:2454] T ceab6b0fccd54e909a64f66fe81a9a1f P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 4.
I20260925 03:04:47.731859 24465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b627ecb4a5384ef392aa695e06599c39" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:47.733042 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 7 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:47.728060 24467 raft_consensus.cc:1261] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260925 03:04:47.736807 24467 raft_consensus.cc:1261] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260925 03:04:47.738718 24466 raft_consensus.cc:1261] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260925 03:04:47.740015 26241 raft_consensus.cc:482] T 367277daf8d04778b3fa4ed94d71580d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:47.740615 26241 raft_consensus.cc:504] T 367277daf8d04778b3fa4ed94d71580d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:47.743228 26241 leader_election.cc:288] T 367277daf8d04778b3fa4ed94d71580d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:47.744288 24616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1414dcfe53f64a78ae413531eb8ec117" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:47.713084 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37a157996e944db2906110d87ecb0c54" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:47.746272 24616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b030d84c670b48d5bff337ccc0c77626" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:47.747956 24616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d89396576884f7c8901dba54f783947" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:47.820650 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1e85248d3b04b979dd080caeeb02fdb" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:47.832930 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15861cd28e354598a3c6bfc4337c34bb" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:47.844237 26357 raft_consensus.cc:482] T ac9357d144aa49789faaf31fe5e6bf89 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:47.844964 26357 raft_consensus.cc:504] T ac9357d144aa49789faaf31fe5e6bf89 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:47.847115 26357 leader_election.cc:288] T ac9357d144aa49789faaf31fe5e6bf89 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:47.848042 24620 raft_consensus.cc:2454] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 5.
I20260925 03:04:47.919132 26529 leader_election.cc:288] T 442d4a29b1d743599f4b6b4871e26668 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:47.937481 26499 raft_consensus.cc:482] T 8e5819c15eac4544a57e5ee6c9d4f117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:47.938299 26499 raft_consensus.cc:504] T 8e5819c15eac4544a57e5ee6c9d4f117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:47.941068 26499 leader_election.cc:288] T 8e5819c15eac4544a57e5ee6c9d4f117 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:47.948848 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b8e0d260a884ecdb20f79de46c36b19" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:47.959053 26619 raft_consensus.cc:504] T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:47.966879 24616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "100c113a9fd84cc8adf2ec7b33ea0093" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:47.968153 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b5df303cd6a4d59aca5c18bbbff24ed" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:47.990518 26437 raft_consensus.cc:482] T c88cbd39e01f44628a094af516be2833 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:47.991034 26621 raft_consensus.cc:482] T 7993f9c3e2a44f408bbf5d1e395e2306 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:47.991700 26630 raft_consensus.cc:482] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:47.996450 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e54ca1757ee4e42956a9cebbeca4f62" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:48.010491 24617 raft_consensus.cc:2454] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 7.
I20260925 03:04:48.011400 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 7 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:47.997535 24618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7aa47c48404465b94a36fd819fe3c7a" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:48.005107 24547 raft_consensus.cc:2454] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 6.
I20260925 03:04:48.008332 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367277daf8d04778b3fa4ed94d71580d" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
W20260925 03:04:48.013041 24547 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 1366) took 1568 ms (client timeout 2043 ms). Trace:
W20260925 03:04:48.013448 24547 rpcz_store.cc:269] 0925 03:04:46.444571 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:47.704867 (+1260296us) service_pool.cc:227] Handling call
0925 03:04:48.012968 (+308101us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:48.014005 24544 raft_consensus.cc:2454] T 420163a6cdce412ebc21c25fe3835638 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 6.
I20260925 03:04:48.014539 24544 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36786 (request call id 887) took 1569 ms. Trace:
I20260925 03:04:48.014398 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c85f5d1d3f004cf5817bb594fd8e6469" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:48.016543 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d89396576884f7c8901dba54f783947" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:48.018319 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b030d84c670b48d5bff337ccc0c77626" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:48.018787 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06f4a7eaf6534e4986c9fd35911038e4" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:48.019073 24543 raft_consensus.cc:2454] T b030d84c670b48d5bff337ccc0c77626 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 6.
I20260925 03:04:48.019477 24622 raft_consensus.cc:3037] T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:04:48.021812 24544 rpcz_store.cc:276] 0925 03:04:46.444961 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:47.708390 (+1263429us) service_pool.cc:227] Handling call
0925 03:04:48.014478 (+306088us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:48.022783 24616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f22ddd7d1c444dc58f45a1fb0b3e02bc" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:48.022851 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74443a0a1dc34d88a2f42e3b722db405" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:48.023370 24471 raft_consensus.cc:2396] T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
W20260925 03:04:48.015874 26628 log.cc:924] Time spent T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5: Append to log took a long time: real 0.327s user 0.001s sys 0.000s
I20260925 03:04:48.003777 26448 consensus_queue.cc:1035] T a7aa47c48404465b94a36fd819fe3c7a P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.021s
I20260925 03:04:48.031392 24548 raft_consensus.cc:1261] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260925 03:04:48.029012 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06f4a7eaf6534e4986c9fd35911038e4" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:48.015161 26437 raft_consensus.cc:504] T c88cbd39e01f44628a094af516be2833 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:48.016274 26621 raft_consensus.cc:504] T 7993f9c3e2a44f408bbf5d1e395e2306 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:48.029340 24539 raft_consensus.cc:1226] T 420163a6cdce412ebc21c25fe3835638 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260925 03:04:48.030211 24618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e725b82985746b39c0cc8f671c95353" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:48.039216 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9c33fd100ff44c1a3247319a324a044" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:48.040083 26619 leader_election.cc:288] T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 election: Requested vote from peers eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:48.032599 24511 leader_election.cc:302] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
W20260925 03:04:48.033954 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 1.41017e+06us
I20260925 03:04:48.041783 26598 raft_consensus.cc:482] T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:48.044009 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e5819c15eac4544a57e5ee6c9d4f117" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:48.053433 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc2df40db08649c9a75e90000b878559" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:48.062606 24615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c85f5d1d3f004cf5817bb594fd8e6469" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:48.018113 26630 raft_consensus.cc:504] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:04:48.163304 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 459943us
I20260925 03:04:48.522080 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b5df303cd6a4d59aca5c18bbbff24ed" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
W20260925 03:04:48.342944 24436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 335272us
I20260925 03:04:48.523648 24548 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 877) took 2105 ms. Trace:
I20260925 03:04:48.072608 24622 raft_consensus.cc:2454] T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 6.
I20260925 03:04:48.524552 25194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1e85248d3b04b979dd080caeeb02fdb" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43"
I20260925 03:04:48.521780 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5332a5d329493991dff994e70ea92c" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:48.007107 26499 raft_consensus.cc:482] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:48.529102 26499 raft_consensus.cc:504] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:48.073226 25193 raft_consensus.cc:1261] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43 [term 7 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260925 03:04:48.532788 25194 raft_consensus.cc:2454] T b1e85248d3b04b979dd080caeeb02fdb P 1bc1f2e62ab44af995165bd5116a9d43 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 5.
I20260925 03:04:48.256969 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7aa47c48404465b94a36fd819fe3c7a" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
W20260925 03:04:48.144256 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 237803us
I20260925 03:04:48.535732 24618 raft_consensus.cc:2454] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 5.
I20260925 03:04:48.537456 26241 raft_consensus.cc:2781] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader election won for term 6
I20260925 03:04:48.524636 24542 raft_consensus.cc:3037] T 8e5819c15eac4544a57e5ee6c9d4f117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Advancing to term 5
I20260925 03:04:48.538026 26241 raft_consensus.cc:686] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 LEADER]: Becoming Leader. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:04:48.538997 26241 consensus_queue.cc:237] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:04:48.540131 24511 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 507596us
I20260925 03:04:48.542241 24511 leader_election.cc:302] T 8e5819c15eac4544a57e5ee6c9d4f117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:48.547421 24544 raft_consensus.cc:2454] T cc2df40db08649c9a75e90000b878559 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 4.
I20260925 03:04:48.549108 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f22ddd7d1c444dc58f45a1fb0b3e02bc" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:48.550117 24544 raft_consensus.cc:2454] T f22ddd7d1c444dc58f45a1fb0b3e02bc P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 5.
I20260925 03:04:48.553212 24514 leader_election.cc:302] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:48.523976 24548 rpcz_store.cc:276] 0925 03:04:46.417957 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:48.030724 (+1612767us) service_pool.cc:227] Handling call
0925 03:04:48.031050 (+ 326us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:48.523399 (+492349us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:48.523598 (+ 199us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:48.535223 24546 raft_consensus.cc:2454] T a7aa47c48404465b94a36fd819fe3c7a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 4.
I20260925 03:04:48.574231 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "293ef69caf514ac98510a68495f54eda" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:48.522886 24547 raft_consensus.cc:2454] T 0b5df303cd6a4d59aca5c18bbbff24ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 6.
I20260925 03:04:48.578346 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bb46bce3564841b7674e6a693bd2e6" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:48.579629 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "100c113a9fd84cc8adf2ec7b33ea0093" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:48.066058 25196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71f461a6e2ff434993b78a2d077f0abd" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:48.580402 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f1fac028a144058b92331c0d9b44ebf" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:48.582476 24543 raft_consensus.cc:2454] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 5.
W20260925 03:04:48.251912 26438 consensus_meta.cc:287] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5: Time spent flushing consensus metadata: real 0.596s user 0.014s sys 0.003s
I20260925 03:04:48.617065 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9c33fd100ff44c1a3247319a324a044" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:48.617852 24548 raft_consensus.cc:2454] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 5.
I20260925 03:04:48.618775 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e725b82985746b39c0cc8f671c95353" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:48.619350 24548 raft_consensus.cc:2454] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 5.
I20260925 03:04:48.613140 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abd5b8dc1b7e49e3aa9af900d73ff0e7" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:48.624104 24547 raft_consensus.cc:2454] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 4.
I20260925 03:04:48.627274 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e725b82985746b39c0cc8f671c95353" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:48.628207 24543 raft_consensus.cc:2379] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 in current term 5: Already voted for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in this term.
I20260925 03:04:48.575121 24546 raft_consensus.cc:2454] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 4.
I20260925 03:04:48.556470 26448 consensus_queue.cc:1225] T a7aa47c48404465b94a36fd819fe3c7a P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer 869b0a8575b54d08ae2817fad5dd42d5 log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260925 03:04:48.633401 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f68804eb5a244d87bb395d50683c6012" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:48.627346 25196 raft_consensus.cc:2379] T 71f461a6e2ff434993b78a2d077f0abd P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c in current term 6: Already voted for candidate 1bc1f2e62ab44af995165bd5116a9d43 in this term.
I20260925 03:04:48.532292 26499 leader_election.cc:288] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:48.635108 24547 raft_consensus.cc:2454] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 3.
I20260925 03:04:48.637511 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceab6b0fccd54e909a64f66fe81a9a1f" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:48.639755 24542 raft_consensus.cc:2454] T 8e5819c15eac4544a57e5ee6c9d4f117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 5.
W20260925 03:04:48.642869 24542 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36786 (request call id 885) took 2219 ms (client timeout 2233 ms). Trace:
I20260925 03:04:48.642915 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71f461a6e2ff434993b78a2d077f0abd" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
W20260925 03:04:48.643301 24542 rpcz_store.cc:269] 0925 03:04:46.423534 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:48.043755 (+1620221us) service_pool.cc:227] Handling call
0925 03:04:48.642816 (+599061us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:48.642918 24544 raft_consensus.cc:2454] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 5.
I20260925 03:04:48.644604 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ac466793d094325ace224708b2158ed" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:48.646576 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5332a5d329493991dff994e70ea92c" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:48.650094 24542 raft_consensus.cc:1261] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260925 03:04:48.651319 24436 leader_election.cc:302] T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5; no voters:
I20260925 03:04:48.653923 24544 raft_consensus.cc:1261] T 0b5df303cd6a4d59aca5c18bbbff24ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260925 03:04:48.555321 26598 raft_consensus.cc:504] T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:48.631453 26438 raft_consensus.cc:504] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:48.650144 24548 raft_consensus.cc:2454] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 5.
W20260925 03:04:48.666199 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 113173us
I20260925 03:04:48.668056 24514 leader_election.cc:302] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:48.673082 26644 raft_consensus.cc:2741] T 8e5819c15eac4544a57e5ee6c9d4f117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260925 03:04:48.675438 26646 raft_consensus.cc:2726] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260925 03:04:48.681124 24514 leader_election.cc:302] T 0be4e0aa984f4f2b81a58e382e434ad5 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:48.684712 26598 leader_election.cc:288] T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
W20260925 03:04:48.685072 24511 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 142895us
W20260925 03:04:48.670622 24643 debug-util.cc:398] Leaking SignalData structure 0x7b0800124180 after lost signal to thread 23300
I20260925 03:04:48.663739 25196 raft_consensus.cc:1261] T 7e49248643df4131a1c42e74244e05c2 P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260925 03:04:48.685876 24511 leader_election.cc:302] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:48.668107 26638 raft_consensus.cc:2781] T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader election won for term 5
I20260925 03:04:48.687378 24514 leader_election.cc:302] T b030d84c670b48d5bff337ccc0c77626 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:48.688591 24511 leader_election.cc:302] T a4abe0ea757e4886be4945c5001e529a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 1bc1f2e62ab44af995165bd5116a9d43, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:48.689118 24514 leader_election.cc:302] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters: 1bc1f2e62ab44af995165bd5116a9d43
I20260925 03:04:48.689668 26648 raft_consensus.cc:2781] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260925 03:04:48.690028 26648 raft_consensus.cc:482] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:48.690379 26648 raft_consensus.cc:3037] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Advancing to term 4
W20260925 03:04:48.686208 24643 debug-util.cc:398] Leaking SignalData structure 0x7b080009d340 after lost signal to thread 23303
W20260925 03:04:48.697121 26633 log.cc:924] Time spent T 0be4e0aa984f4f2b81a58e382e434ad5 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.063s user 0.000s sys 0.000s
I20260925 03:04:48.685442 26639 raft_consensus.cc:482] T 25102ba08e4e49388f6176badcd10a25 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:48.698045 26639 raft_consensus.cc:504] T 25102ba08e4e49388f6176badcd10a25 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:48.698949 26438 leader_election.cc:288] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 8 election: Requested vote from peers eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:48.700225 26639 leader_election.cc:288] T 25102ba08e4e49388f6176badcd10a25 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:48.701005 26437 leader_election.cc:288] T c88cbd39e01f44628a094af516be2833 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:48.676738 26646 raft_consensus.cc:2726] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260925 03:04:48.704527 26598 consensus_queue.cc:1035] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260925 03:04:48.705139 25196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5aec38e66a947c681e6c2a72b05e6ca" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 8 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43"
I20260925 03:04:48.704885 26644 raft_consensus.cc:2726] T b030d84c670b48d5bff337ccc0c77626 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260925 03:04:48.689275 26643 raft_consensus.cc:2726] T 0be4e0aa984f4f2b81a58e382e434ad5 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260925 03:04:48.746343 24511 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.4:43979, user_credentials={real_user=slave}} blocked reactor thread for 43711.4us
I20260925 03:04:48.703064 26638 raft_consensus.cc:686] T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 LEADER]: Becoming Leader. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:04:48.745223 26646 raft_consensus.cc:2726] T a4abe0ea757e4886be4945c5001e529a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260925 03:04:48.747731 25196 raft_consensus.cc:3037] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43 [term 7 FOLLOWER]: Advancing to term 8
I20260925 03:04:48.757988 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e5819c15eac4544a57e5ee6c9d4f117" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:48.759030 24547 raft_consensus.cc:2454] T 8e5819c15eac4544a57e5ee6c9d4f117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 5.
I20260925 03:04:48.759527 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60d45b7d467d4e1ab92367f6d7d52b15" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:48.760175 24542 raft_consensus.cc:2454] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 6.
I20260925 03:04:48.760538 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c23337a7cb74184a0e57b7095cb6a11" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:48.761015 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48ca6ccd98194cf2af7b5278d15bf4ea" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:48.761193 24544 raft_consensus.cc:2454] T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 5.
I20260925 03:04:48.761591 24543 raft_consensus.cc:2454] T 48ca6ccd98194cf2af7b5278d15bf4ea P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 3.
I20260925 03:04:48.762367 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35992316cb8d4c028333388eff02ecde" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:48.763166 24546 raft_consensus.cc:2396] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 for term 7 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260925 03:04:48.764779 26646 consensus_queue.cc:1046] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20260925 03:04:48.765762 26632 log.cc:924] Time spent T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.111s user 0.000s sys 0.000s
I20260925 03:04:48.766808 24619 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38570 (request call id 1402) took 3389 ms. Trace:
I20260925 03:04:48.767171 24619 rpcz_store.cc:276] 0925 03:04:45.377612 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:45.386594 (+ 8982us) service_pool.cc:227] Handling call
0925 03:04:45.386933 (+ 339us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:45.573155 (+186222us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:48.766576 (+3193421us) spinlock_profiling.cc:243] Waited 185 ms on lock 0x7b54002fd5d4. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d0f51 00007f43764cf99e 00007f437b4112e9 00007f4374af7684 00007f4374af7612 00007f4374af7541 00007f4374af74bc 00007f4374af6772 00007f43748fa94c 00007f43748f9d86
0925 03:04:48.766752 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":426460544}
I20260925 03:04:48.752074 26643 raft_consensus.cc:2741] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20260925 03:04:48.769892 24548 raft_consensus.cc:1261] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260925 03:04:48.770272 24471 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 1036) took 1047 ms. Trace:
I20260925 03:04:48.770632 24471 rpcz_store.cc:276] 0925 03:04:47.723308 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:47.820283 (+ 96975us) service_pool.cc:227] Handling call
0925 03:04:48.770087 (+949804us) spinlock_profiling.cc:243] Waited 201 ms on lock 0x7b540018fe54. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d42ae 00007f437b411c27 00007f4374af9c04 00007f4374af9b92 00007f4374af9ac1 00007f4374af9a3c 00007f4374af8cf2 00007f43748fa94c 00007f43748f9d86 00007f43748f972f
0925 03:04:48.770220 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":463436160}
I20260925 03:04:48.759572 24437 leader_election.cc:302] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:04:48.773250 24437 leader_election.cc:302] T 7b8e0d260a884ecdb20f79de46c36b19 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:04:48.775446 26638 consensus_queue.cc:237] T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:48.778170 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c88cbd39e01f44628a094af516be2833" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:48.778820 24546 raft_consensus.cc:2454] T c88cbd39e01f44628a094af516be2833 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 5.
I20260925 03:04:48.779769 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5f9b503b0ab43999d582015a367441e" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:48.780169 24539 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 1378) took 1079 ms. Trace:
I20260925 03:04:48.780385 24543 raft_consensus.cc:2454] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 6.
W20260925 03:04:48.788878 24620 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38570 (request call id 1419) took 2388 ms (client timeout 1849 ms). Trace:
W20260925 03:04:48.789292 24620 rpcz_store.cc:269] 0925 03:04:46.400781 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:46.420001 (+ 19220us) service_pool.cc:227] Handling call
0925 03:04:48.788705 (+2368704us) spinlock_profiling.cc:243] Waited 1.41 s on lock 0x7b5400249d54. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d42ae 00007f437b411c27 00007f4374af9c04 00007f4374af9b92 00007f4374af9ac1 00007f4374af9a3c 00007f4374af8cf2 00007f43748fa94c 00007f43748f9d86 00007f43748f972f
0925 03:04:48.788811 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3235181440}
I20260925 03:04:48.791961 25199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ac466793d094325ace224708b2158ed" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:48.792483 25199 raft_consensus.cc:2454] T 8ac466793d094325ace224708b2158ed P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 4.
W20260925 03:04:48.793165 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 1.08833e+06us
I20260925 03:04:48.793880 24588 leader_election.cc:302] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:48.765704 26646 raft_consensus.cc:3032] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 LEADER]: Stepping down as leader of term 5
W20260925 03:04:48.801085 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 112043us
I20260925 03:04:48.801375 24588 leader_election.cc:302] T 25102ba08e4e49388f6176badcd10a25 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:48.802006 24437 leader_election.cc:302] T 550118bc1a0641ce9803bdbf43f2069f P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:04:48.805796 24514 leader_election.cc:302] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:48.805801 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25102ba08e4e49388f6176badcd10a25" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:48.807821 24437 leader_election.cc:302] T edcf6e6a422d4d198d4fbb0f7d3d845d P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
W20260925 03:04:48.805605 24585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.4:43979, user_credentials={real_user=slave}} blocked reactor thread for 802926us
I20260925 03:04:48.810453 24514 leader_election.cc:302] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5
W20260925 03:04:48.810504 24586 connection.cc:232] RPC call timeout handler was delayed by -1.4176819469994371s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20990000000000003s for a response
I20260925 03:04:48.808955 24548 raft_consensus.cc:2454] T 25102ba08e4e49388f6176badcd10a25 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 3.
I20260925 03:04:48.799546 26639 raft_consensus.cc:2781] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260925 03:04:48.804330 26366 raft_consensus.cc:2781] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Leader election won for term 5
I20260925 03:04:48.819262 24437 leader_election.cc:302] T 1414dcfe53f64a78ae413531eb8ec117 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:04:48.804932 26646 raft_consensus.cc:729] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:04:48.805161 26630 leader_election.cc:288] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:48.799929 26642 consensus_queue.cc:1035] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.046s
W20260925 03:04:48.835146 24586 connection.cc:232] RPC call timeout handler was delayed by -1.3480138409995561s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21530000000000002s for a response
I20260925 03:04:48.805469 26438 raft_consensus.cc:2781] T 7b8e0d260a884ecdb20f79de46c36b19 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260925 03:04:48.836127 24588 leader_election.cc:302] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:04:48.867475 24585 leader_election.cc:302] T 04e068a0f80941dc9138cee063f53bab P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:48.871420 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ac466793d094325ace224708b2158ed" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:48.873802 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e92f377a79354a6ebb4eec67303c13e3" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:48.872349 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71f461a6e2ff434993b78a2d077f0abd" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:48.874562 24542 raft_consensus.cc:2454] T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 4.
W20260925 03:04:48.875039 24586 connection.cc:232] RPC call timeout handler was delayed by -1.3204958999995142s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.193s for a response
I20260925 03:04:48.875839 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6b6ec010caf4689b8c90449764f5cf9" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
W20260925 03:04:48.876319 24586 connection.cc:232] RPC call timeout handler was delayed by -1.2583505660004448s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.189s for a response
W20260925 03:04:48.876839 24586 connection.cc:232] RPC call timeout handler was delayed by -1.2099711730006675s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.213s for a response
I20260925 03:04:48.815490 26643 raft_consensus.cc:2726] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260925 03:04:48.877826 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71f461a6e2ff434993b78a2d077f0abd" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:48.878360 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4abe0ea757e4886be4945c5001e529a" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:48.879451 24585 leader_election.cc:302] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:48.815055 26644 raft_consensus.cc:2726] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260925 03:04:48.814563 26637 raft_consensus.cc:2781] T 550118bc1a0641ce9803bdbf43f2069f P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260925 03:04:48.870460 26598 raft_consensus.cc:2781] T 1414dcfe53f64a78ae413531eb8ec117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260925 03:04:48.850497 26649 raft_consensus.cc:482] T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:48.842265 26437 raft_consensus.cc:2781] T edcf6e6a422d4d198d4fbb0f7d3d845d P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260925 03:04:48.887907 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367277daf8d04778b3fa4ed94d71580d" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:48.840767 26552 raft_consensus.cc:2781] T 25102ba08e4e49388f6176badcd10a25 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader election won for term 3
I20260925 03:04:48.896930 24544 raft_consensus.cc:2454] T 367277daf8d04778b3fa4ed94d71580d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 4.
I20260925 03:04:48.872498 26650 raft_consensus.cc:482] T 5278a60f97da4e10aecd50a8d9a9a43c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
W20260925 03:04:48.880548 26634 log.cc:924] Time spent T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.191s user 0.000s sys 0.001s
I20260925 03:04:48.818091 26639 raft_consensus.cc:482] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:48.897450 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 7 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:48.871160 26438 raft_consensus.cc:482] T 7b8e0d260a884ecdb20f79de46c36b19 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:48.901540 24546 raft_consensus.cc:2454] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 6.
I20260925 03:04:48.797844 24539 rpcz_store.cc:276] 0925 03:04:47.700589 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:47.728290 (+ 27701us) service_pool.cc:227] Handling call
0925 03:04:48.016106 (+287816us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:04:48.533552 (+517446us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:48.779972 (+246420us) spinlock_profiling.cc:243] Waited 286 ms on lock 0x7b540017aa50. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764cfa09 00007f437b4112e9 00007f4374af7684 00007f4374af7612 00007f4374af7541 00007f4374af74bc 00007f4374af6772 00007f43748fa94c 00007f43748f9d86 00007f43748f972f
0925 03:04:48.780115 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":658688256}
I20260925 03:04:48.896018 26637 raft_consensus.cc:482] T 550118bc1a0641ce9803bdbf43f2069f P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:48.889031 26380 raft_consensus.cc:482] T 21c43af5e6844fdaacc8c625fa01fd8d P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:48.895413 26598 raft_consensus.cc:482] T 1414dcfe53f64a78ae413531eb8ec117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:48.919749 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15861cd28e354598a3c6bfc4337c34bb" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:48.903126 26270 raft_consensus.cc:2726] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260925 03:04:48.896517 26437 raft_consensus.cc:482] T edcf6e6a422d4d198d4fbb0f7d3d845d P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:48.904323 26438 raft_consensus.cc:3037] T 7b8e0d260a884ecdb20f79de46c36b19 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:48.894315 26551 raft_consensus.cc:2781] T 04e068a0f80941dc9138cee063f53bab P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260925 03:04:48.902837 26639 raft_consensus.cc:3037] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:04:48.914966 26386 consensus_queue.cc:1035] T 7993f9c3e2a44f408bbf5d1e395e2306 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.019s
I20260925 03:04:48.914824 26552 raft_consensus.cc:686] T 25102ba08e4e49388f6176badcd10a25 P eeab93abfdb141a895002ec9f312296c [term 3 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
W20260925 03:04:48.932057 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 112862us
I20260925 03:04:48.901806 26563 consensus_queue.cc:1035] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.025s
I20260925 03:04:48.933753 24588 leader_election.cc:302] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:48.921639 26637 raft_consensus.cc:3037] T 550118bc1a0641ce9803bdbf43f2069f P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Advancing to term 5
W20260925 03:04:48.934773 24586 connection.cc:232] RPC call timeout handler was delayed by -1.0001848469983088s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22330000000000003s for a response
W20260925 03:04:48.923545 24585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.4:43979, user_credentials={real_user=slave}} blocked reactor thread for 44182.1us
I20260925 03:04:48.921169 26448 raft_consensus.cc:482] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:48.949410 26448 raft_consensus.cc:504] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:48.924093 26380 raft_consensus.cc:504] T 21c43af5e6844fdaacc8c625fa01fd8d P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:48.952666 26293 raft_consensus.cc:2781] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260925 03:04:48.953215 26293 raft_consensus.cc:482] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:48.953548 26293 raft_consensus.cc:3037] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:04:48.954355 24614 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38570 (request call id 1423) took 1267 ms. Trace:
I20260925 03:04:48.954722 24614 rpcz_store.cc:276] 0925 03:04:47.687407 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:47.749484 (+ 62077us) service_pool.cc:227] Handling call
0925 03:04:47.749813 (+ 329us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:04:48.793929 (+1044116us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:48.793961 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:04:48.794419 (+ 458us) log.cc:842] Serialized 26 byte log entry
0925 03:04:48.886119 (+ 91700us) raft_consensus.cc:1631] Marking committed up to 4
0925 03:04:48.886162 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:48.886234 (+ 72us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:04:48.954143 (+ 67909us) raft_consensus.cc:1671] finished
0925 03:04:48.954164 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:48.954283 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1043701,"raft.queue_time_us":1044771,"raft.run_cpu_time_us":165,"raft.run_wall_time_us":5405,"thread_start_us":91358,"threads_started":1,"wal-append.queue_time_us":121748}
I20260925 03:04:48.904137 26650 raft_consensus.cc:504] T 5278a60f97da4e10aecd50a8d9a9a43c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:04:48.956604 24545 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33068 (request call id 114) took 4321 ms (client timeout 1516 ms). Trace:
W20260925 03:04:48.957053 24545 rpcz_store.cc:269] 0925 03:04:44.635626 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:44.737898 (+102272us) service_pool.cc:227] Handling call
0925 03:04:48.956403 (+4218505us) spinlock_profiling.cc:243] Waited 123 ms on lock 0x7b540010de54. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d42ae 00007f437b411c27 00007f4374af9c04 00007f4374af9b92 00007f4374af9ac1 00007f4374af9a3c 00007f4374af8cf2 00007f43748fa94c 00007f43748f9d86 00007f43748f972f
0925 03:04:48.956512 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":282541312}
I20260925 03:04:48.925792 26437 raft_consensus.cc:3037] T edcf6e6a422d4d198d4fbb0f7d3d845d P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:48.938690 24469 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38920 (request call id 123) took 1258 ms. Trace:
I20260925 03:04:48.961313 24469 rpcz_store.cc:276] 0925 03:04:47.679959 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:48.933807 (+1253848us) service_pool.cc:227] Handling call
0925 03:04:48.938208 (+ 4401us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:48.938390 (+ 182us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:48.938415 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:48.938454 (+ 39us) raft_consensus.cc:1631] Marking committed up to 7
0925 03:04:48.938481 (+ 27us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:48.938531 (+ 50us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:48.938632 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:48.936766 26386 consensus_queue.cc:1225] T 7993f9c3e2a44f408bbf5d1e395e2306 P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer 869b0a8575b54d08ae2817fad5dd42d5 log is divergent from this leader: its last log entry 4.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260925 03:04:48.906301 26646 consensus_queue.cc:260] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:48.939956 26648 raft_consensus.cc:504] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:48.977051 26552 consensus_queue.cc:237] T 25102ba08e4e49388f6176badcd10a25 P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:48.901540 26649 raft_consensus.cc:504] T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:48.937533 26366 raft_consensus.cc:686] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c [term 5 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:48.924654 26544 raft_consensus.cc:2781] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260925 03:04:48.981498 26544 raft_consensus.cc:482] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting leader election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:48.981942 26544 raft_consensus.cc:3037] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:48.951475 26448 leader_election.cc:288] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:48.922384 26598 raft_consensus.cc:3037] T 1414dcfe53f64a78ae413531eb8ec117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:48.936612 24437 leader_election.cc:302] T 37a157996e944db2906110d87ecb0c54 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:04:48.964227 24541 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 1368) took 1269 ms. Trace:
I20260925 03:04:48.965242 25162 leader_election.cc:302] T 29637f5a4a504ce6ba29033ccd0cd716 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:48.969718 25196 raft_consensus.cc:2454] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 8.
I20260925 03:04:48.935280 26551 raft_consensus.cc:482] T 04e068a0f80941dc9138cee063f53bab P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1bc1f2e62ab44af995165bd5116a9d43)
I20260925 03:04:48.974116 26380 leader_election.cc:288] T 21c43af5e6844fdaacc8c625fa01fd8d P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:48.973834 26245 consensus_queue.cc:1035] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20260925 03:04:48.980630 26648 leader_election.cc:288] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:48.988126 26650 leader_election.cc:288] T 5278a60f97da4e10aecd50a8d9a9a43c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:48.988844 26646 raft_consensus.cc:3037] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:04:49.027482 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74443a0a1dc34d88a2f42e3b722db405" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:48.990727 24621 raft_consensus.cc:2454] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 6.
W20260925 03:04:48.991677 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 55136.5us
I20260925 03:04:48.991520 24541 rpcz_store.cc:276] 0925 03:04:47.694609 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:47.721764 (+ 27155us) service_pool.cc:227] Handling call
0925 03:04:47.722129 (+ 365us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:04:47.722472 (+ 343us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:47.722500 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:04:47.744180 (+ 21680us) log.cc:842] Serialized 26 byte log entry
0925 03:04:48.144520 (+400340us) raft_consensus.cc:1631] Marking committed up to 7
0925 03:04:48.267177 (+122657us) log.cc:842] Serialized 14 byte log entry
0925 03:04:48.267424 (+ 247us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:48.267501 (+ 77us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:04:48.964015 (+696514us) raft_consensus.cc:1671] finished
0925 03:04:48.964037 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:48.964169 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":122921,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":47,"raft.queue_time_us":42899,"raft.run_cpu_time_us":282,"raft.run_wall_time_us":53035,"thread_start_us":399927,"threads_started":1,"wal-append.queue_time_us":917887}
I20260925 03:04:49.029042 26247 raft_consensus.cc:482] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260925 03:04:49.029114 24621 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38570 (request call id 1418) took 2628 ms (client timeout 3015 ms). Trace:
I20260925 03:04:48.993865 26209 raft_consensus.cc:2781] T 37a157996e944db2906110d87ecb0c54 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260925 03:04:48.994334 26649 leader_election.cc:288] T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:48.994869 26448 consensus_queue.cc:1035] T 0b5df303cd6a4d59aca5c18bbbff24ed P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260925 03:04:48.996289 24542 raft_consensus.cc:1226] T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Rejecting Update request from peer 1bc1f2e62ab44af995165bd5116a9d43 for earlier term 2. Current term is 3. Ops: []
I20260925 03:04:48.997287 24540 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 864) took 4070 ms. Trace:
I20260925 03:04:49.029101 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bb46bce3564841b7674e6a693bd2e6" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:49.030241 26209 raft_consensus.cc:482] T 37a157996e944db2906110d87ecb0c54 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:49.030476 24437 leader_election.cc:302] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:04:49.031019 26209 raft_consensus.cc:3037] T 37a157996e944db2906110d87ecb0c54 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Advancing to term 6
W20260925 03:04:49.030017 24621 rpcz_store.cc:269] 0925 03:04:46.400515 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:46.435458 (+ 34943us) service_pool.cc:227] Handling call
0925 03:04:49.029056 (+2593598us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:48.997954 26637 raft_consensus.cc:504] T 550118bc1a0641ce9803bdbf43f2069f P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:49.032194 26642 raft_consensus.cc:2781] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260925 03:04:49.032572 26642 raft_consensus.cc:482] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:49.032974 26642 raft_consensus.cc:3037] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:49.033330 26637 leader_election.cc:288] T 550118bc1a0641ce9803bdbf43f2069f P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:48.999653 26450 raft_consensus.cc:2726] T 29637f5a4a504ce6ba29033ccd0cd716 P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260925 03:04:49.001469 26438 raft_consensus.cc:504] T 7b8e0d260a884ecdb20f79de46c36b19 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:04:49.002170 25163 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 1.372s user 0.022s sys 0.014s
I20260925 03:04:49.003435 26598 raft_consensus.cc:504] T 1414dcfe53f64a78ae413531eb8ec117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:48.999560 26608 consensus_queue.cc:1035] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.006s
I20260925 03:04:49.034220 26247 raft_consensus.cc:504] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:49.007758 26621 leader_election.cc:288] T 7993f9c3e2a44f408bbf5d1e395e2306 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:49.008905 26386 consensus_queue.cc:1035] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260925 03:04:49.036708 26598 leader_election.cc:288] T 1414dcfe53f64a78ae413531eb8ec117 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:49.037132 24437 leader_election.cc:302] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:04:49.030493 24540 rpcz_store.cc:276] 0925 03:04:44.926775 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:45.498678 (+571903us) service_pool.cc:227] Handling call
0925 03:04:48.957962 (+3459284us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:48.958141 (+ 179us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:48.958167 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:48.958210 (+ 43us) raft_consensus.cc:1631] Marking committed up to 5
0925 03:04:48.958239 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:48.958294 (+ 55us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:48.997120 (+ 38826us) spinlock_profiling.cc:243] Waited 3.45 s on lock 0x7b540010de50. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764cfa09 00007f437b4112e9 00007f4374af7684 00007f4374af7612 00007f4374af7541 00007f4374af74bc 00007f4374af6772 00007f43748fa94c 00007f43748f9d86 00007f43748f972f
0925 03:04:48.997227 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7938097408}
W20260925 03:04:49.038247 24643 debug-util.cc:398] Leaking SignalData structure 0x7b08000c5580 after lost signal to thread 24405
I20260925 03:04:49.007618 26544 raft_consensus.cc:504] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:49.013339 26551 raft_consensus.cc:3037] T 04e068a0f80941dc9138cee063f53bab P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:49.016096 26656 consensus_queue.cc:1035] T 550118bc1a0641ce9803bdbf43f2069f P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260925 03:04:49.018306 24586 leader_election.cc:302] T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
W20260925 03:04:49.014343 26607 consensus_meta.cc:287] T 48ca6ccd98194cf2af7b5278d15bf4ea P eeab93abfdb141a895002ec9f312296c: Time spent flushing consensus metadata: real 1.351s user 0.005s sys 0.032s
I20260925 03:04:49.054683 24586 leader_election.cc:302] T 43366a0a2f9d4344bf4e37ffd290ef8a P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:04:49.055564 26552 raft_consensus.cc:2781] T 43366a0a2f9d4344bf4e37ffd290ef8a P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader election won for term 3
W20260925 03:04:49.021613 24436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 67778us
I20260925 03:04:49.039213 26438 leader_election.cc:288] T 7b8e0d260a884ecdb20f79de46c36b19 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:49.040640 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc2df40db08649c9a75e90000b878559" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:49.057092 25199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e54ca1757ee4e42956a9cebbeca4f62" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:49.057951 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceab6b0fccd54e909a64f66fe81a9a1f" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:49.041602 24437 leader_election.cc:302] T cc2df40db08649c9a75e90000b878559 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:04:49.041749 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abd5b8dc1b7e49e3aa9af900d73ff0e7" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:49.010749 26293 raft_consensus.cc:504] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:49.059190 24436 leader_election.cc:302] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5; no voters:
I20260925 03:04:49.059546 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c88cbd39e01f44628a094af516be2833" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:49.060559 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25102ba08e4e49388f6176badcd10a25" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:49.061511 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5f9b503b0ab43999d582015a367441e" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:49.062511 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "550118bc1a0641ce9803bdbf43f2069f" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:49.062986 24586 leader_election.cc:302] T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:04:49.063472 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1414dcfe53f64a78ae413531eb8ec117" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:49.045943 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e5819c15eac4544a57e5ee6c9d4f117" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:49.065402 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7993f9c3e2a44f408bbf5d1e395e2306" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:49.066143 26293 leader_election.cc:288] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 6 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:49.038352 24371 catalog_manager.cc:5495] T e92f377a79354a6ebb4eec67303c13e3 P 1bc1f2e62ab44af995165bd5116a9d43 reported cstate change: term changed from 1 to 3, leader changed from eeab93abfdb141a895002ec9f312296c (127.22.193.3) to 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4). New cstate: current_term: 3 leader_uuid: "1bc1f2e62ab44af995165bd5116a9d43" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260925 03:04:49.066246 26386 consensus_queue.cc:1035] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260925 03:04:49.048950 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f1fac028a144058b92331c0d9b44ebf" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:49.050514 24614 raft_consensus.cc:1261] T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260925 03:04:49.068439 26438 raft_consensus.cc:2781] T cc2df40db08649c9a75e90000b878559 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260925 03:04:49.051713 25163 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.422s user 0.023s sys 0.015s
I20260925 03:04:49.068836 26438 raft_consensus.cc:482] T cc2df40db08649c9a75e90000b878559 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
W20260925 03:04:49.069080 25163 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 1.43923e+06us
I20260925 03:04:49.069205 26438 raft_consensus.cc:3037] T cc2df40db08649c9a75e90000b878559 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:49.069660 25196 raft_consensus.cc:1226] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43 [term 8 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 7. Current term is 8. Ops: [5.6-7.7]
W20260925 03:04:49.070041 25163 leader_election.cc:334] T e6b6ec010caf4689b8c90449764f5cf9 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389): Timed out: RequestConsensusVote RPC to 127.22.193.2:36389 timed out after 1.831s (SENT)
I20260925 03:04:49.070448 25163 leader_election.cc:302] T e6b6ec010caf4689b8c90449764f5cf9 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:49.068125 24437 leader_election.cc:302] T f22ddd7d1c444dc58f45a1fb0b3e02bc P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:04:49.019570 26577 leader_election.cc:288] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:49.047742 24616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c23337a7cb74184a0e57b7095cb6a11" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:49.071475 24437 leader_election.cc:302] T a7aa47c48404465b94a36fd819fe3c7a P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:04:49.067003 24621 raft_consensus.cc:2354] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 5. Current term is 6.
I20260925 03:04:49.061445 26378 consensus_queue.cc:1035] T a7aa47c48404465b94a36fd819fe3c7a P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.006s
I20260925 03:04:49.066378 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5aec38e66a947c681e6c2a72b05e6ca" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 8 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:49.073608 26621 raft_consensus.cc:2781] T f22ddd7d1c444dc58f45a1fb0b3e02bc P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260925 03:04:49.075426 26621 raft_consensus.cc:482] T f22ddd7d1c444dc58f45a1fb0b3e02bc P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:49.075776 26621 raft_consensus.cc:3037] T f22ddd7d1c444dc58f45a1fb0b3e02bc P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:04:49.053668 26508 raft_consensus.cc:2726] T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260925 03:04:49.077498 23300 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260925 03:04:49.078182 23300 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260925 03:04:49.064433 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b8e0d260a884ecdb20f79de46c36b19" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:49.079455 24436 leader_election.cc:302] T 3d89396576884f7c8901dba54f783947 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:49.080425 26637 raft_consensus.cc:2781] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Leader election won for term 8
W20260925 03:04:49.039741 24643 debug-util.cc:398] Leaking SignalData structure 0x7b08001ee5e0 after lost signal to thread 24418
I20260925 03:04:49.080897 26637 raft_consensus.cc:686] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 LEADER]: Becoming Leader. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
W20260925 03:04:49.082126 24643 debug-util.cc:398] Leaking SignalData structure 0x7b0800315540 after lost signal to thread 24497
I20260925 03:04:49.081735 26637 consensus_queue.cc:237] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:49.077425 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ac466793d094325ace224708b2158ed" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:49.082285 26450 raft_consensus.cc:2726] T e6b6ec010caf4689b8c90449764f5cf9 P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260925 03:04:49.084400 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48ca6ccd98194cf2af7b5278d15bf4ea" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:49.084869 26640 raft_consensus.cc:2726] T 3d89396576884f7c8901dba54f783947 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260925 03:04:49.085178 26642 raft_consensus.cc:504] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:49.086598 24436 leader_election.cc:302] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters: 1bc1f2e62ab44af995165bd5116a9d43
I20260925 03:04:49.049355 26649 raft_consensus.cc:482] T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:49.050539 26655 raft_consensus.cc:2781] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260925 03:04:49.092914 26655 raft_consensus.cc:482] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:49.093331 26655 raft_consensus.cc:3037] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:04:49.086997 26544 leader_election.cc:288] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:49.084003 26247 leader_election.cc:288] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:49.075191 26209 raft_consensus.cc:504] T 37a157996e944db2906110d87ecb0c54 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:49.095820 26438 raft_consensus.cc:504] T cc2df40db08649c9a75e90000b878559 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:49.096586 26209 leader_election.cc:288] T 37a157996e944db2906110d87ecb0c54 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:49.097762 26438 leader_election.cc:288] T cc2df40db08649c9a75e90000b878559 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:49.098115 26241 raft_consensus.cc:482] T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:49.098654 26241 raft_consensus.cc:504] T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:49.101748 24436 leader_election.cc:302] T 06f4a7eaf6534e4986c9fd35911038e4 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5
W20260925 03:04:49.082465 24643 debug-util.cc:398] Leaking SignalData structure 0x7b0800251200 after lost signal to thread 24498
I20260925 03:04:49.104555 24436 leader_election.cc:302] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:49.104584 24586 leader_election.cc:302] T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:49.105767 24436 leader_election.cc:302] T c85f5d1d3f004cf5817bb594fd8e6469 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:49.088601 24437 leader_election.cc:302] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:04:49.071753 24615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1e85248d3b04b979dd080caeeb02fdb" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:49.092249 26640 raft_consensus.cc:2781] T a7aa47c48404465b94a36fd819fe3c7a P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260925 03:04:49.092047 26649 raft_consensus.cc:504] T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:49.117051 24615 raft_consensus.cc:2379] T b1e85248d3b04b979dd080caeeb02fdb P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 in current term 5: Already voted for candidate eeab93abfdb141a895002ec9f312296c in this term.
I20260925 03:04:49.121858 24437 leader_election.cc:302] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:04:49.111761 24371 catalog_manager.cc:5495] T a4abe0ea757e4886be4945c5001e529a P 1bc1f2e62ab44af995165bd5116a9d43 reported cstate change: term changed from 1 to 2, leader changed from a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2) to eeab93abfdb141a895002ec9f312296c (127.22.193.3). New cstate: current_term: 2 leader_uuid: "eeab93abfdb141a895002ec9f312296c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:49.120666 26438 raft_consensus.cc:2781] T 06f4a7eaf6534e4986c9fd35911038e4 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader election won for term 6
I20260925 03:04:49.125262 24436 leader_election.cc:302] T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:49.087036 26642 leader_election.cc:288] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:49.100823 26386 raft_consensus.cc:2781] T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260925 03:04:49.089308 26552 raft_consensus.cc:686] T 43366a0a2f9d4344bf4e37ffd290ef8a P eeab93abfdb141a895002ec9f312296c [term 3 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
W20260925 03:04:49.125662 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 135351us
I20260925 03:04:49.126744 24437 leader_election.cc:302] T 100c113a9fd84cc8adf2ec7b33ea0093 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:49.100364 26638 raft_consensus.cc:2781] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260925 03:04:49.110064 24643 debug-util.cc:398] Leaking SignalData structure 0x7b08002b0ca0 after lost signal to thread 24572
W20260925 03:04:49.127842 25163 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 46068us
I20260925 03:04:49.130802 24437 leader_election.cc:302] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:04:49.108335 26241 leader_election.cc:288] T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:49.123415 26598 raft_consensus.cc:2781] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260925 03:04:49.115669 26640 raft_consensus.cc:482] T a7aa47c48404465b94a36fd819fe3c7a P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:49.073009 26639 raft_consensus.cc:504] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:49.072880 26607 raft_consensus.cc:504] T 48ca6ccd98194cf2af7b5278d15bf4ea P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:49.088650 26437 raft_consensus.cc:504] T edcf6e6a422d4d198d4fbb0f7d3d845d P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:49.126116 26544 consensus_queue.cc:1035] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.016s
I20260925 03:04:49.142391 26246 raft_consensus.cc:2726] T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260925 03:04:49.107319 26619 raft_consensus.cc:2726] T c85f5d1d3f004cf5817bb594fd8e6469 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260925 03:04:49.145593 24586 leader_election.cc:302] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5
W20260925 03:04:49.132673 24643 debug-util.cc:398] Leaking SignalData structure 0x7b08002fada0 after lost signal to thread 24573
I20260925 03:04:49.124791 26209 raft_consensus.cc:2781] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260925 03:04:49.108923 26371 consensus_queue.cc:1046] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.027s
I20260925 03:04:49.128471 26386 raft_consensus.cc:482] T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Starting leader election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:49.128880 26499 raft_consensus.cc:2781] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260925 03:04:49.145056 26598 raft_consensus.cc:482] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:49.153951 25163 leader_election.cc:302] T 367277daf8d04778b3fa4ed94d71580d P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:04:49.136613 24588 leader_election.cc:302] T b1e85248d3b04b979dd080caeeb02fdb P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:04:49.131981 26450 raft_consensus.cc:482] T a4abe0ea757e4886be4945c5001e529a P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
W20260925 03:04:49.147027 26660 log.cc:924] Time spent T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.056s user 0.001s sys 0.000s
I20260925 03:04:49.160321 24586 leader_election.cc:302] T 420163a6cdce412ebc21c25fe3835638 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:04:49.132916 26638 raft_consensus.cc:482] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 1bc1f2e62ab44af995165bd5116a9d43)
I20260925 03:04:49.162814 26637 raft_consensus.cc:2726] T 100c113a9fd84cc8adf2ec7b33ea0093 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260925 03:04:49.156630 26437 leader_election.cc:288] T edcf6e6a422d4d198d4fbb0f7d3d845d P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:49.146711 26640 raft_consensus.cc:3037] T a7aa47c48404465b94a36fd819fe3c7a P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:49.170074 24437 leader_election.cc:302] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:04:49.147113 25196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ac466793d094325ace224708b2158ed" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43"
I20260925 03:04:49.132499 26247 consensus_queue.cc:1046] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.026s
I20260925 03:04:49.182236 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f68804eb5a244d87bb395d50683c6012" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:49.184216 24471 raft_consensus.cc:1226] T edcf6e6a422d4d198d4fbb0f7d3d845d P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260925 03:04:49.186846 24466 raft_consensus.cc:1226] T 7b8e0d260a884ecdb20f79de46c36b19 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 4. Current term is 5. Ops: []
I20260925 03:04:49.198621 24437 leader_election.cc:302] T 293ef69caf514ac98510a68495f54eda P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 1bc1f2e62ab44af995165bd5116a9d43
I20260925 03:04:49.199193 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e8af2641e124f9589c5ccfdac785f63" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:49.200439 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21c43af5e6844fdaacc8c625fa01fd8d" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:49.104507 26366 consensus_queue.cc:237] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:04:49.191749 24436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 66573.7us
I20260925 03:04:49.204454 24437 leader_election.cc:302] T 4e5332a5d329493991dff994e70ea92c P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
W20260925 03:04:49.192286 24367 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39368 (request call id 196) took 13353 ms (client timeout 15000 ms). Trace:
I20260925 03:04:49.132652 26642 raft_consensus.cc:2726] T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260925 03:04:49.168655 26209 raft_consensus.cc:482] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:49.170714 26386 raft_consensus.cc:3037] T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:04:49.153285 26401 consensus_queue.cc:1046] T b4ebd2b043024a548005c8b6b00b0449 P 1bc1f2e62ab44af995165bd5116a9d43 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.044s
I20260925 03:04:49.203357 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e725b82985746b39c0cc8f671c95353" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:49.153178 26400 consensus_queue.cc:1035] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.038s
I20260925 03:04:49.168716 26245 raft_consensus.cc:2781] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260925 03:04:49.204620 24436 leader_election.cc:302] T 11bb46bce3564841b7674e6a693bd2e6 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:49.153322 26509 consensus_queue.cc:1035] T 7993f9c3e2a44f408bbf5d1e395e2306 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.039s
I20260925 03:04:49.204457 24468 raft_consensus.cc:1226] T 550118bc1a0641ce9803bdbf43f2069f P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 4. Current term is 5. Ops: []
I20260925 03:04:49.203617 24470 raft_consensus.cc:1226] T 1414dcfe53f64a78ae413531eb8ec117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 2. Current term is 3. Ops: []
I20260925 03:04:49.206645 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5332a5d329493991dff994e70ea92c" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:49.207093 24622 raft_consensus.cc:1226] T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 5. Current term is 6. Ops: []
I20260925 03:04:49.207345 24472 raft_consensus.cc:2379] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c in current term 5: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:04:49.208568 24471 raft_consensus.cc:1226] T 37a157996e944db2906110d87ecb0c54 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 5. Current term is 6. Ops: []
I20260925 03:04:49.176122 26598 raft_consensus.cc:3037] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:49.208909 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b627ecb4a5384ef392aa695e06599c39" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:49.209996 24616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ac285175464fbda67211d9574db415" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:49.170928 26499 raft_consensus.cc:482] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:49.192117 26619 raft_consensus.cc:2781] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260925 03:04:49.231720 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7993f9c3e2a44f408bbf5d1e395e2306" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:49.212602 24586 leader_election.cc:302] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:49.212771 24614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367277daf8d04778b3fa4ed94d71580d" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:49.174702 26438 raft_consensus.cc:686] T 06f4a7eaf6534e4986c9fd35911038e4 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 LEADER]: Becoming Leader. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:04:49.193284 26608 raft_consensus.cc:2726] T b1e85248d3b04b979dd080caeeb02fdb P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260925 03:04:49.207998 24613 raft_consensus.cc:2379] T 48ca6ccd98194cf2af7b5278d15bf4ea P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 in current term 4: Already voted for candidate eeab93abfdb141a895002ec9f312296c in this term.
I20260925 03:04:49.231715 24618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc2df40db08649c9a75e90000b878559" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:49.194676 26638 raft_consensus.cc:3037] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Advancing to term 8
I20260925 03:04:49.231536 24615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b8e0d260a884ecdb20f79de46c36b19" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:49.235653 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442d4a29b1d743599f4b6b4871e26668" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:49.207551 26577 raft_consensus.cc:2781] T 420163a6cdce412ebc21c25fe3835638 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Leader election won for term 6
I20260925 03:04:49.238312 24616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f68804eb5a244d87bb395d50683c6012" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:49.239204 24586 leader_election.cc:302] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:49.238692 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e54ca1757ee4e42956a9cebbeca4f62" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:49.244946 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9357d144aa49789faaf31fe5e6bf89" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
W20260925 03:04:49.234697 24643 debug-util.cc:398] Leaking SignalData structure 0x7b08003b4360 after lost signal to thread 24646
I20260925 03:04:49.207527 26637 raft_consensus.cc:3032] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 LEADER]: Stepping down as leader of term 5
I20260925 03:04:49.251855 26649 leader_election.cc:288] T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:49.252209 26637 raft_consensus.cc:729] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:04:49.253074 26637 consensus_queue.cc:260] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:49.254285 26637 raft_consensus.cc:3037] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:04:49.241997 26544 raft_consensus.cc:2781] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260925 03:04:49.234377 26378 consensus_queue.cc:1035] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.023s
I20260925 03:04:49.257169 26544 raft_consensus.cc:482] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:49.257560 26544 raft_consensus.cc:3037] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:04:49.260334 26630 raft_consensus.cc:2781] T 4e5332a5d329493991dff994e70ea92c P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260925 03:04:49.236981 26642 raft_consensus.cc:2781] T 293ef69caf514ac98510a68495f54eda P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260925 03:04:49.237620 26619 raft_consensus.cc:482] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:49.240082 26245 raft_consensus.cc:482] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Starting leader election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:49.261838 26245 raft_consensus.cc:3037] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:04:49.263145 26607 leader_election.cc:288] T 48ca6ccd98194cf2af7b5278d15bf4ea P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:49.190775 26450 raft_consensus.cc:504] T a4abe0ea757e4886be4945c5001e529a P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:49.263731 26655 raft_consensus.cc:504] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:49.264904 26671 raft_consensus.cc:482] T 04e068a0f80941dc9138cee063f53bab P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:49.265411 26450 leader_election.cc:288] T a4abe0ea757e4886be4945c5001e529a P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
W20260925 03:04:49.265918 26661 log.cc:924] Time spent T 550118bc1a0641ce9803bdbf43f2069f P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.108s user 0.000s sys 0.000s
W20260925 03:04:49.252012 24643 debug-util.cc:398] Leaking SignalData structure 0x7b08003b4340 after lost signal to thread 24647
I20260925 03:04:49.237085 26670 raft_consensus.cc:482] T b4ebd2b043024a548005c8b6b00b0449 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1bc1f2e62ab44af995165bd5116a9d43)
I20260925 03:04:49.266975 26670 raft_consensus.cc:504] T b4ebd2b043024a548005c8b6b00b0449 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
W20260925 03:04:49.267491 24643 debug-util.cc:398] Leaking SignalData structure 0x7b08001a8f60 after lost signal to thread 25223
I20260925 03:04:49.266136 24616 raft_consensus.cc:2454] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 4.
I20260925 03:04:49.237231 26672 raft_consensus.cc:2781] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260925 03:04:49.268987 26672 raft_consensus.cc:482] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:49.269356 26672 raft_consensus.cc:3037] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:49.271505 26621 raft_consensus.cc:504] T f22ddd7d1c444dc58f45a1fb0b3e02bc P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:49.207641 25196 raft_consensus.cc:3037] T 8ac466793d094325ace224708b2158ed P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:49.213304 26209 raft_consensus.cc:3037] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:04:49.262233 26639 leader_election.cc:288] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:49.249207 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5278a60f97da4e10aecd50a8d9a9a43c" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
W20260925 03:04:49.205235 24367 rpcz_store.cc:269] 0925 03:04:35.839163 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:36.712887 (+873724us) service_pool.cc:227] Handling call
0925 03:04:37.883078 (+1170191us) spinlock_profiling.cc:243] Waited 46.2 ms on lock 0x7b5400002728. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f437c859501 00007f437c8545bc 00007f437c42ad54 00007f437c524c3e 00007f437a836fb4 00007f437a836f42 00007f437a836e71 00007f437a836dec 00007f437a8360a2 00007f43748fa94c 00007f43748f9d86
0925 03:04:49.192233 (+11309155us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0925 03:04:41.546255 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0925 03:04:41.676219 (+129964us) write_op.cc:432] Acquiring schema lock in shared mode
0925 03:04:41.676276 (+ 57us) write_op.cc:435] Acquired schema lock
0925 03:04:41.676287 (+ 11us) tablet.cc:660] Decoding operations
0925 03:04:41.737040 (+ 60753us) write_op.cc:620] Acquiring the partition lock for write op
0925 03:04:41.737141 (+ 101us) write_op.cc:641] Partition lock acquired for write op
0925 03:04:41.737187 (+ 46us) tablet.cc:683] Acquiring locks for 16 operations
0925 03:04:41.738408 (+ 1221us) tablet.cc:699] Row locks acquired
0925 03:04:41.738446 (+ 38us) write_op.cc:260] PREPARE: finished
0925 03:04:41.738611 (+ 165us) write_op.cc:270] Start()
0925 03:04:41.738762 (+ 151us) write_op.cc:276] Timestamp: P: 1790305481738552 usec, L: 0
0925 03:04:41.738798 (+ 36us) op_driver.cc:331] REPLICATION: starting
0925 03:04:41.783921 (+ 45123us) log.cc:842] Serialized 6365 byte log entry
0925 03:04:42.717816 (+933895us) op_driver.cc:444] REPLICATION: finished
0925 03:04:43.046292 (+328476us) write_op.cc:301] APPLY: starting
0925 03:04:43.059644 (+ 13352us) tablet.cc:1365] starting BulkCheckPresence
0925 03:04:43.188928 (+129284us) tablet.cc:1368] finished BulkCheckPresence
0925 03:04:43.189090 (+ 162us) tablet.cc:1370] starting ApplyRowOperation cycle
0925 03:04:43.237814 (+ 48724us) tablet.cc:1381] finished ApplyRowOperation cycle
0925 03:04:43.238152 (+ 338us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=16
0925 03:04:43.238215 (+ 63us) write_op.cc:312] APPLY: finished
0925 03:04:43.238709 (+ 494us) log.cc:842] Serialized 112 byte log entry
0925 03:04:43.238885 (+ 176us) write_op.cc:489] Releasing partition, row and schema locks
0925 03:04:43.239062 (+ 177us) write_op.cc:454] Released schema lock
0925 03:04:43.239124 (+ 62us) write_op.cc:341] FINISH: Updating metrics
0925 03:04:43.239394 (+ 270us) write_op.cc:489] Releasing partition, row and schema locks
0925 03:04:43.239420 (+ 26us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1085443,"reported_tablets":37,"spinlock_wait_cycles":106303872,"child_traces":[["op",{"apply.queue_time_us":178651,"apply.run_cpu_time_us":5973,"apply.run_wall_time_us":262512,"num_ops":16,"prepare.queue_time_us":228268,"prepare.run_cpu_time_us":20609,"prepare.run_wall_time_us":403049,"raft.queue_time_us":332204,"raft.run_cpu_time_us":34332,"raft.run_wall_time_us":272247,"replication_time_us":921507,"thread_start_us":460893,"threads_started":4,"wal-append.queue_time_us":122496,"wal-append.run_cpu_time_us":26269,"wal-append.run_wall_time_us":2307038}]]}
I20260925 03:04:49.260900 26630 raft_consensus.cc:482] T 4e5332a5d329493991dff994e70ea92c P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
W20260925 03:04:49.261502 26662 log.cc:924] Time spent T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.108s user 0.000s sys 0.000s
I20260925 03:04:49.364568 24616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e068a0f80941dc9138cee063f53bab" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
W20260925 03:04:49.267800 24643 debug-util.cc:398] Leaking SignalData structure 0x7b080019df20 after lost signal to thread 25224
I20260925 03:04:49.235204 26499 raft_consensus.cc:3037] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:04:49.367327 24467 raft_consensus.cc:1226] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 6. Current term is 7. Ops: []
W20260925 03:04:49.274260 26660 log.cc:924] Time spent T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.059s user 0.000s sys 0.000s
I20260925 03:04:49.364498 24472 pending_rounds.cc:76] T 7993f9c3e2a44f408bbf5d1e395e2306 P 869b0a8575b54d08ae2817fad5dd42d5: Aborting all ops after (but not including) 1
W20260925 03:04:49.369745 25163 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 215879us
I20260925 03:04:49.273766 26619 raft_consensus.cc:3037] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Advancing to term 6
W20260925 03:04:49.373155 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 168783us
I20260925 03:04:49.382891 24437 leader_election.cc:302] T 8e5819c15eac4544a57e5ee6c9d4f117 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:04:49.367432 24471 raft_consensus.cc:1226] T f22ddd7d1c444dc58f45a1fb0b3e02bc P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 5. Current term is 6. Ops: []
I20260925 03:04:49.375878 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74443a0a1dc34d88a2f42e3b722db405" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:49.376245 26677 raft_consensus.cc:3032] T b4ebd2b043024a548005c8b6b00b0449 P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 LEADER]: Stepping down as leader of term 2
I20260925 03:04:49.380491 24472 pending_rounds.cc:97] T 7993f9c3e2a44f408bbf5d1e395e2306 P 869b0a8575b54d08ae2817fad5dd42d5: Aborting uncommitted NO_OP operation due to leader change: 4.2
I20260925 03:04:49.266014 26450 raft_consensus.cc:2781] T 367277daf8d04778b3fa4ed94d71580d P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260925 03:04:49.384788 26677 raft_consensus.cc:729] T b4ebd2b043024a548005c8b6b00b0449 P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1bc1f2e62ab44af995165bd5116a9d43, State: Running, Role: LEADER
I20260925 03:04:49.384876 24472 raft_consensus.cc:2866] T 7993f9c3e2a44f408bbf5d1e395e2306 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260925 03:04:49.374717 25162 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 199684us
I20260925 03:04:49.383487 24616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bb46bce3564841b7674e6a693bd2e6" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:49.385475 26450 raft_consensus.cc:482] T 367277daf8d04778b3fa4ed94d71580d P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:49.386904 26450 raft_consensus.cc:3037] T 367277daf8d04778b3fa4ed94d71580d P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:49.381596 25163 leader_election.cc:302] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:49.386336 26677 consensus_queue.cc:260] T b4ebd2b043024a548005c8b6b00b0449 P 1bc1f2e62ab44af995165bd5116a9d43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:49.390161 24616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4abe0ea757e4886be4945c5001e529a" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:49.261016 26642 raft_consensus.cc:482] T 293ef69caf514ac98510a68495f54eda P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 1bc1f2e62ab44af995165bd5116a9d43)
I20260925 03:04:49.391207 25163 leader_election.cc:302] T 8ac466793d094325ace224708b2158ed P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:49.197494 26552 consensus_queue.cc:237] T 43366a0a2f9d4344bf4e37ffd290ef8a P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:49.394721 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceab6b0fccd54e909a64f66fe81a9a1f" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:49.396373 24468 raft_consensus.cc:3037] T ceab6b0fccd54e909a64f66fe81a9a1f P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Advancing to term 6
I20260925 03:04:49.383847 24437 leader_election.cc:302] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:04:49.395087 24586 leader_election.cc:302] T 71f461a6e2ff434993b78a2d077f0abd P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 1bc1f2e62ab44af995165bd5116a9d43, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:49.371686 26608 raft_consensus.cc:2781] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260925 03:04:49.363471 26241 raft_consensus.cc:482] T 21c43af5e6844fdaacc8c625fa01fd8d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:49.398295 24616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e8af2641e124f9589c5ccfdac785f63" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:49.372622 26669 raft_consensus.cc:2726] T 11bb46bce3564841b7674e6a693bd2e6 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260925 03:04:49.403507 25199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4ebd2b043024a548005c8b6b00b0449" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:49.405534 24511 leader_election.cc:302] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:49.374032 26621 leader_election.cc:288] T f22ddd7d1c444dc58f45a1fb0b3e02bc P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:49.374670 26630 raft_consensus.cc:3037] T 4e5332a5d329493991dff994e70ea92c P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:04:49.374876 26577 raft_consensus.cc:686] T 420163a6cdce412ebc21c25fe3835638 P eeab93abfdb141a895002ec9f312296c [term 6 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:49.380388 26648 raft_consensus.cc:482] T 1414dcfe53f64a78ae413531eb8ec117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:49.391844 26677 raft_consensus.cc:3037] T b4ebd2b043024a548005c8b6b00b0449 P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:49.265362 26671 raft_consensus.cc:504] T 04e068a0f80941dc9138cee063f53bab P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:49.432149 24616 raft_consensus.cc:1226] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 4. Current term is 5. Ops: []
I20260925 03:04:49.394371 26401 raft_consensus.cc:2726] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260925 03:04:49.422976 24468 raft_consensus.cc:2454] T ceab6b0fccd54e909a64f66fe81a9a1f P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 6.
I20260925 03:04:49.395804 26642 raft_consensus.cc:3037] T 293ef69caf514ac98510a68495f54eda P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:49.401464 26608 raft_consensus.cc:482] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:49.444805 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7691c6b475428c82d6181d693d26f3" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:49.424975 26664 raft_consensus.cc:3037] T 71f461a6e2ff434993b78a2d077f0abd P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:04:49.461709 24468 raft_consensus.cc:2379] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c in current term 7: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:04:49.414958 26670 leader_election.cc:288] T b4ebd2b043024a548005c8b6b00b0449 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:49.424662 26530 raft_consensus.cc:2781] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader election won for term 4
I20260925 03:04:49.411976 25193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15861cd28e354598a3c6bfc4337c34bb" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:49.461385 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71f461a6e2ff434993b78a2d077f0abd" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:49.501396 24542 raft_consensus.cc:2396] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1bc1f2e62ab44af995165bd5116a9d43 for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260925 03:04:49.502199 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7691c6b475428c82d6181d693d26f3" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:49.466722 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4abe0ea757e4886be4945c5001e529a" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:49.467404 25191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e068a0f80941dc9138cee063f53bab" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:49.429811 26621 raft_consensus.cc:2781] T 8e5819c15eac4544a57e5ee6c9d4f117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260925 03:04:49.417404 26366 consensus_queue.cc:1035] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.046s
I20260925 03:04:49.507097 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25102ba08e4e49388f6176badcd10a25" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:49.441066 26378 consensus_queue.cc:1035] T 0b5df303cd6a4d59aca5c18bbbff24ed P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.047s
I20260925 03:04:49.384183 26245 raft_consensus.cc:504] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:49.488675 25194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367277daf8d04778b3fa4ed94d71580d" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:49.496315 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e8af2641e124f9589c5ccfdac785f63" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:49.460048 26650 consensus_queue.cc:1046] T 8ac466793d094325ace224708b2158ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.037s
I20260925 03:04:49.504123 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21c43af5e6844fdaacc8c625fa01fd8d" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:49.509691 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ac466793d094325ace224708b2158ed" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:49.510895 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71f461a6e2ff434993b78a2d077f0abd" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:49.408092 26241 raft_consensus.cc:504] T 21c43af5e6844fdaacc8c625fa01fd8d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:49.514532 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceab6b0fccd54e909a64f66fe81a9a1f" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:49.514828 25163 leader_election.cc:302] T a4abe0ea757e4886be4945c5001e529a P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:49.528981 24541 raft_consensus.cc:3037] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:04:49.507808 24546 raft_consensus.cc:2362] T 25102ba08e4e49388f6176badcd10a25 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 3. Re-sending same reply.
W20260925 03:04:49.516661 24436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 273166us
I20260925 03:04:49.465103 26669 raft_consensus.cc:482] T ac9357d144aa49789faaf31fe5e6bf89 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:49.418462 26644 consensus_queue.cc:1046] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.094s
I20260925 03:04:49.535938 24436 leader_election.cc:398] T ceab6b0fccd54e909a64f66fe81a9a1f P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 pre-election: Vote denied by peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105) with higher term. Message: Invalid argument: T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 5. Current term is 6.
I20260925 03:04:49.536357 24436 leader_election.cc:401] T ceab6b0fccd54e909a64f66fe81a9a1f P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20260925 03:04:49.440287 26648 raft_consensus.cc:504] T 1414dcfe53f64a78ae413531eb8ec117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:49.525715 24540 raft_consensus.cc:2396] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 1bc1f2e62ab44af995165bd5116a9d43 for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260925 03:04:49.525930 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48ca6ccd98194cf2af7b5278d15bf4ea" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:49.537492 24542 raft_consensus.cc:3037] T 48ca6ccd98194cf2af7b5278d15bf4ea P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:49.464190 26608 raft_consensus.cc:3037] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:49.415426 26438 consensus_queue.cc:237] T 06f4a7eaf6534e4986c9fd35911038e4 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:04:49.544536 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 160885us
I20260925 03:04:49.512260 26621 raft_consensus.cc:482] T 8e5819c15eac4544a57e5ee6c9d4f117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:49.545485 24437 leader_election.cc:302] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:04:49.548388 25163 leader_election.cc:302] T 71f461a6e2ff434993b78a2d077f0abd P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:49.550734 24544 raft_consensus.cc:2454] T 21c43af5e6844fdaacc8c625fa01fd8d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 3.
I20260925 03:04:49.560590 24541 raft_consensus.cc:2454] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 6.
I20260925 03:04:49.563884 24437 leader_election.cc:302] T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:04:49.532174 24539 raft_consensus.cc:2454] T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 4.
I20260925 03:04:49.564867 24437 leader_election.cc:302] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:49.519129 26530 raft_consensus.cc:686] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 LEADER]: Becoming Leader. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:04:49.525642 25191 raft_consensus.cc:2454] T 04e068a0f80941dc9138cee063f53bab P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 3.
I20260925 03:04:49.539249 26646 raft_consensus.cc:3032] T 8ac466793d094325ace224708b2158ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 LEADER]: Stepping down as leader of term 4
I20260925 03:04:49.530902 26401 raft_consensus.cc:2726] T a4abe0ea757e4886be4945c5001e529a P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260925 03:04:49.560525 24542 raft_consensus.cc:2454] T 48ca6ccd98194cf2af7b5278d15bf4ea P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 4.
I20260925 03:04:49.550588 26621 raft_consensus.cc:3037] T 8e5819c15eac4544a57e5ee6c9d4f117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:04:49.562547 26650 raft_consensus.cc:3032] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 LEADER]: Stepping down as leader of term 5
I20260925 03:04:49.513235 26577 consensus_queue.cc:237] T 420163a6cdce412ebc21c25fe3835638 P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:49.572805 26655 leader_election.cc:288] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
W20260925 03:04:49.643739 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 90223us
I20260925 03:04:49.714082 24586 leader_election.cc:302] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:49.544451 26669 raft_consensus.cc:504] T ac9357d144aa49789faaf31fe5e6bf89 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:04:49.643424 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 78643us
I20260925 03:04:49.548286 26670 consensus_queue.cc:1035] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.086s
I20260925 03:04:49.567268 26679 raft_consensus.cc:2726] T 71f461a6e2ff434993b78a2d077f0abd P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260925 03:04:49.572503 26686 raft_consensus.cc:2726] T ceab6b0fccd54e909a64f66fe81a9a1f P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105) with higher term. Message: Invalid argument: T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 5. Current term is 6.
W20260925 03:04:49.700766 26675 log.cc:924] Time spent T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5: Append to log took a long time: real 0.085s user 0.001s sys 0.000s
I20260925 03:04:49.634831 26682 raft_consensus.cc:2781] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260925 03:04:49.638136 26646 raft_consensus.cc:729] T 8ac466793d094325ace224708b2158ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:04:49.647367 26371 raft_consensus.cc:2781] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260925 03:04:49.646472 26687 raft_consensus.cc:2781] T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260925 03:04:49.739409 26686 consensus_queue.cc:1035] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260925 03:04:49.741231 26509 raft_consensus.cc:2781] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Leader election won for term 6
I20260925 03:04:49.757866 26509 raft_consensus.cc:686] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [term 6 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:49.758807 26509 consensus_queue.cc:237] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:49.742624 26293 raft_consensus.cc:3032] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [term 7 LEADER]: Stepping down as leader of term 7
I20260925 03:04:49.742278 26530 consensus_queue.cc:237] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:49.760852 26293 raft_consensus.cc:729] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [term 7 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:49.700489 26650 raft_consensus.cc:729] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:04:49.750032 26644 raft_consensus.cc:482] T 37a157996e944db2906110d87ecb0c54 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:49.736953 24437 leader_election.cc:302] T c88cbd39e01f44628a094af516be2833 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:04:49.769632 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48ca6ccd98194cf2af7b5278d15bf4ea" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:49.748034 26371 raft_consensus.cc:482] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:49.785050 25191 raft_consensus.cc:1226] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43 [term 8 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 7. Current term is 8. Ops: [5.6-7.7]
I20260925 03:04:49.795013 26371 raft_consensus.cc:3037] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Advancing to term 6
W20260925 03:04:49.786042 26698 log.cc:924] Time spent T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.065s user 0.000s sys 0.000s
I20260925 03:04:49.788776 26646 consensus_queue.cc:260] T 8ac466793d094325ace224708b2158ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:49.787585 26650 consensus_queue.cc:260] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:49.798511 26650 raft_consensus.cc:3037] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:04:49.776628 26644 raft_consensus.cc:504] T 37a157996e944db2906110d87ecb0c54 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:49.751650 26687 raft_consensus.cc:482] T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:49.801158 26687 raft_consensus.cc:3037] T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:49.784358 26686 raft_consensus.cc:2781] T c88cbd39e01f44628a094af516be2833 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260925 03:04:49.788930 24472 raft_consensus.cc:1226] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 LEADER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 7. Current term is 8. Ops: [5.6-7.7]
I20260925 03:04:49.803804 26686 raft_consensus.cc:482] T c88cbd39e01f44628a094af516be2833 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:49.804236 26686 raft_consensus.cc:3037] T c88cbd39e01f44628a094af516be2833 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:04:49.768193 26293 consensus_queue.cc:260] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
W20260925 03:04:49.816808 26688 log.cc:924] Time spent T 00000000000000000000000000000000 P 2d92cc0732c4490eb7d98d353826cce1: Append to log took a long time: real 0.094s user 0.000s sys 0.004s
W20260925 03:04:49.817070 26246 consensus_queue.cc:1159] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "1bc1f2e62ab44af995165bd5116a9d43" responder_term: 8 status { last_received { term: 3 index: 5 } last_committed_idx: 5 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 7. Current term is 8. Ops: [5.6-7.7]" } } last_received_current_leader { term: 0 index: 0 } }
W20260925 03:04:49.879513 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 142670us
I20260925 03:04:49.880470 24437 leader_election.cc:302] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:04:49.880785 24467 raft_consensus.cc:1226] T cc2df40db08649c9a75e90000b878559 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 4. Current term is 5. Ops: []
I20260925 03:04:49.881810 24437 leader_election.cc:302] T 25102ba08e4e49388f6176badcd10a25 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:04:49.698623 24511 leader_election.cc:302] T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
W20260925 03:04:49.892246 24511 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.4:43979, user_credentials={real_user=slave}} blocked reactor thread for 193702us
W20260925 03:04:49.921162 26598 consensus_meta.cc:287] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5: Time spent flushing consensus metadata: real 0.682s user 0.002s sys 0.012s
I20260925 03:04:49.924806 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceab6b0fccd54e909a64f66fe81a9a1f" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:49.930796 24548 raft_consensus.cc:2354] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 5. Current term is 6.
W20260925 03:04:49.909307 26544 consensus_meta.cc:287] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c: Time spent flushing consensus metadata: real 0.651s user 0.011s sys 0.007s
I20260925 03:04:49.913434 26649 raft_consensus.cc:2781] T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260925 03:04:49.957736 24436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 402543us
I20260925 03:04:49.908970 26690 raft_consensus.cc:2781] T 25102ba08e4e49388f6176badcd10a25 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260925 03:04:49.957736 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "550118bc1a0641ce9803bdbf43f2069f" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:49.964869 24614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f22ddd7d1c444dc58f45a1fb0b3e02bc" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:49.964377 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37a157996e944db2906110d87ecb0c54" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:49.912475 26692 raft_consensus.cc:482] T 43366a0a2f9d4344bf4e37ffd290ef8a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:49.970468 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc2df40db08649c9a75e90000b878559" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:49.961982 24616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f68804eb5a244d87bb395d50683c6012" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:49.964681 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7691c6b475428c82d6181d693d26f3" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:49.975287 24621 raft_consensus.cc:3037] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:04:49.979035 24618 raft_consensus.cc:1226] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 6. Current term is 7. Ops: []
I20260925 03:04:49.984028 24615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcf6e6a422d4d198d4fbb0f7d3d845d" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:49.984313 25198 raft_consensus.cc:1261] T b1e85248d3b04b979dd080caeeb02fdb P 1bc1f2e62ab44af995165bd5116a9d43 [term 5 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260925 03:04:49.984543 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc2df40db08649c9a75e90000b878559" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:49.985549 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1414dcfe53f64a78ae413531eb8ec117" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:49.961534 26649 raft_consensus.cc:482] T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:49.968281 26690 raft_consensus.cc:482] T 25102ba08e4e49388f6176badcd10a25 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:49.993299 24621 raft_consensus.cc:2454] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 7.
I20260925 03:04:49.984247 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b030d84c670b48d5bff337ccc0c77626" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:50.003780 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b030d84c670b48d5bff337ccc0c77626" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:49.953223 26544 raft_consensus.cc:504] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:49.980074 26644 leader_election.cc:288] T 37a157996e944db2906110d87ecb0c54 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:49.952724 26598 raft_consensus.cc:504] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:50.028218 24615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e068a0f80941dc9138cee063f53bab" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:49.935834 26448 consensus_queue.cc:1046] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.064s
W20260925 03:04:50.012655 26637 consensus_meta.cc:287] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5: Time spent flushing consensus metadata: real 0.758s user 0.010s sys 0.010s
W20260925 03:04:50.021719 24466 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.579s user 0.001s sys 0.013s
I20260925 03:04:49.987128 26639 raft_consensus.cc:482] T 43366a0a2f9d4344bf4e37ffd290ef8a P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:50.034737 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f22ddd7d1c444dc58f45a1fb0b3e02bc" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:50.117373 24546 raft_consensus.cc:3037] T f22ddd7d1c444dc58f45a1fb0b3e02bc P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:04:50.000872 26648 leader_election.cc:288] T 1414dcfe53f64a78ae413531eb8ec117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:50.001133 26695 raft_consensus.cc:482] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
W20260925 03:04:50.030429 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 316436us
I20260925 03:04:50.001088 26649 raft_consensus.cc:3037] T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Advancing to term 4
W20260925 03:04:50.033282 26551 consensus_meta.cc:287] T 04e068a0f80941dc9138cee063f53bab P eeab93abfdb141a895002ec9f312296c: Time spent flushing consensus metadata: real 0.962s user 0.019s sys 0.000s
I20260925 03:04:50.010294 26241 leader_election.cc:288] T 21c43af5e6844fdaacc8c625fa01fd8d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:50.009428 26671 leader_election.cc:288] T 04e068a0f80941dc9138cee063f53bab P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:50.008301 26669 leader_election.cc:288] T ac9357d144aa49789faaf31fe5e6bf89 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:50.008601 26690 raft_consensus.cc:3037] T 25102ba08e4e49388f6176badcd10a25 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:49.986351 26689 raft_consensus.cc:2726] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260925 03:04:50.115329 24466 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.673s user 0.003s sys 0.016s
I20260925 03:04:50.130440 24586 leader_election.cc:302] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5
W20260925 03:04:50.031366 26450 consensus_meta.cc:287] T 367277daf8d04778b3fa4ed94d71580d P 1bc1f2e62ab44af995165bd5116a9d43: Time spent flushing consensus metadata: real 0.644s user 0.007s sys 0.019s
I20260925 03:04:50.012017 25190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4ebd2b043024a548005c8b6b00b0449" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:49.984315 26656 consensus_queue.cc:1046] T edcf6e6a422d4d198d4fbb0f7d3d845d P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.089s
I20260925 03:04:50.158206 24469 raft_consensus.cc:2454] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 5.
I20260925 03:04:50.156960 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b8e0d260a884ecdb20f79de46c36b19" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
W20260925 03:04:50.180032 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 186414us
I20260925 03:04:50.063118 26706 raft_consensus.cc:482] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:49.973385 26692 raft_consensus.cc:504] T 43366a0a2f9d4344bf4e37ffd290ef8a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:50.063442 26245 leader_election.cc:288] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 7 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:50.197145 24613 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38570 (request call id 1456) took 1144 ms. Trace:
I20260925 03:04:50.183085 26648 raft_consensus.cc:482] T 420163a6cdce412ebc21c25fe3835638 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:50.004871 26545 consensus_queue.cc:1046] T 550118bc1a0641ce9803bdbf43f2069f P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 1, Time since last communication: 0.126s
I20260925 03:04:50.000190 26552 consensus_queue.cc:1046] T 37a157996e944db2906110d87ecb0c54 P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.130s
W20260925 03:04:50.217064 26640 consensus_meta.cc:287] T a7aa47c48404465b94a36fd819fe3c7a P 869b0a8575b54d08ae2817fad5dd42d5: Time spent flushing consensus metadata: real 1.009s user 0.007s sys 0.008s
I20260925 03:04:50.076763 26366 consensus_queue.cc:1046] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.107s
I20260925 03:04:50.212639 24588 leader_election.cc:302] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:50.218698 23300 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260925 03:04:50.245848 24546 raft_consensus.cc:2454] T f22ddd7d1c444dc58f45a1fb0b3e02bc P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 6.
I20260925 03:04:50.222537 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:50.251554 24544 raft_consensus.cc:2362] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 6. Re-sending same reply.
I20260925 03:04:50.224869 24613 rpcz_store.cc:276] 0925 03:04:49.053147 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:49.053783 (+ 636us) service_pool.cc:227] Handling call
0925 03:04:50.196928 (+1143145us) spinlock_profiling.cc:243] Waited 76.8 ms on lock 0x7b54000bf4d4. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d42ae 00007f437b411c27 00007f4374af9c04 00007f4374af9b92 00007f4374af9ac1 00007f4374af9a3c 00007f4374af8cf2 00007f43748fa94c 00007f43748f9d86 00007f43748f972f
0925 03:04:50.197073 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":176596864}
I20260925 03:04:50.251876 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e068a0f80941dc9138cee063f53bab" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
W20260925 03:04:50.198271 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 67938us
I20260925 03:04:50.253386 24586 leader_election.cc:302] T 48ca6ccd98194cf2af7b5278d15bf4ea P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
W20260925 03:04:50.214854 26630 consensus_meta.cc:287] T 4e5332a5d329493991dff994e70ea92c P 869b0a8575b54d08ae2817fad5dd42d5: Time spent flushing consensus metadata: real 0.773s user 0.007s sys 0.011s
W20260925 03:04:50.216972 26642 consensus_meta.cc:287] T 293ef69caf514ac98510a68495f54eda P 869b0a8575b54d08ae2817fad5dd42d5: Time spent flushing consensus metadata: real 0.751s user 0.013s sys 0.001s
I20260925 03:04:50.012529 26696 consensus_queue.cc:1046] T 1414dcfe53f64a78ae413531eb8ec117 P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.175s
I20260925 03:04:50.126690 26639 raft_consensus.cc:504] T 43366a0a2f9d4344bf4e37ffd290ef8a P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:04:50.219432 26386 consensus_meta.cc:287] T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c: Time spent flushing consensus metadata: real 0.982s user 0.013s sys 0.000s
W20260925 03:04:50.258723 26707 log.cc:924] Time spent T 43366a0a2f9d4344bf4e37ffd290ef8a P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.063s user 0.000s sys 0.000s
W20260925 03:04:50.232669 26672 consensus_meta.cc:287] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P 869b0a8575b54d08ae2817fad5dd42d5: Time spent flushing consensus metadata: real 0.963s user 0.009s sys 0.010s
W20260925 03:04:50.260493 25196 consensus_meta.cc:287] T 8ac466793d094325ace224708b2158ed P 1bc1f2e62ab44af995165bd5116a9d43: Time spent flushing consensus metadata: real 0.886s user 0.007s sys 0.011s
I20260925 03:04:50.262671 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4ebd2b043024a548005c8b6b00b0449" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:50.298980 24544 raft_consensus.cc:2379] T b4ebd2b043024a548005c8b6b00b0449 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 in current term 3: Already voted for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in this term.
I20260925 03:04:50.180575 26551 raft_consensus.cc:504] T 04e068a0f80941dc9138cee063f53bab P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:50.230069 26448 raft_consensus.cc:2781] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260925 03:04:50.265789 26679 raft_consensus.cc:482] T b1e85248d3b04b979dd080caeeb02fdb P 1bc1f2e62ab44af995165bd5116a9d43 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:50.215324 26450 raft_consensus.cc:504] T 367277daf8d04778b3fa4ed94d71580d P 1bc1f2e62ab44af995165bd5116a9d43 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:50.328433 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7993f9c3e2a44f408bbf5d1e395e2306" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:50.241518 26648 raft_consensus.cc:504] T 420163a6cdce412ebc21c25fe3835638 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:50.291106 26719 raft_consensus.cc:482] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:50.292778 26671 raft_consensus.cc:482] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:50.345811 24615 raft_consensus.cc:2454] T 04e068a0f80941dc9138cee063f53bab P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 3.
I20260925 03:04:50.347914 24437 leader_election.cc:398] T ceab6b0fccd54e909a64f66fe81a9a1f P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 pre-election: Vote denied by peer a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389) with higher term. Message: Invalid argument: T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 5. Current term is 6.
I20260925 03:04:50.348820 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37a157996e944db2906110d87ecb0c54" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:50.290832 26380 raft_consensus.cc:2726] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260925 03:04:50.354295 24544 raft_consensus.cc:3037] T 37a157996e944db2906110d87ecb0c54 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:04:50.362102 25197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43"
I20260925 03:04:50.250097 26640 raft_consensus.cc:504] T a7aa47c48404465b94a36fd819fe3c7a P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:50.356051 24437 leader_election.cc:302] T 550118bc1a0641ce9803bdbf43f2069f P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
W20260925 03:04:50.358242 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 62575.1us
W20260925 03:04:50.372572 24514 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.22.193.1:42187 timed out after 1.556s (SENT) [suppressed 12 similar messages]
I20260925 03:04:50.327708 26448 raft_consensus.cc:482] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting leader election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:50.282831 26642 raft_consensus.cc:504] T 293ef69caf514ac98510a68495f54eda P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:50.302026 25196 raft_consensus.cc:2454] T 8ac466793d094325ace224708b2158ed P 1bc1f2e62ab44af995165bd5116a9d43 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 5.
I20260925 03:04:50.277253 26630 raft_consensus.cc:504] T 4e5332a5d329493991dff994e70ea92c P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:50.342959 26607 raft_consensus.cc:2781] T 48ca6ccd98194cf2af7b5278d15bf4ea P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader election won for term 4
I20260925 03:04:50.346786 24466 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1060) took 1156 ms. Trace:
I20260925 03:04:50.368961 25194 raft_consensus.cc:2379] T 367277daf8d04778b3fa4ed94d71580d P 1bc1f2e62ab44af995165bd5116a9d43 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 5: Already voted for candidate 1bc1f2e62ab44af995165bd5116a9d43 in this term.
I20260925 03:04:50.294149 26386 raft_consensus.cc:504] T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:50.382032 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcf6e6a422d4d198d4fbb0f7d3d845d" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
W20260925 03:04:50.377632 26677 consensus_meta.cc:287] T b4ebd2b043024a548005c8b6b00b0449 P 1bc1f2e62ab44af995165bd5116a9d43: Time spent flushing consensus metadata: real 0.937s user 0.018s sys 0.002s
I20260925 03:04:50.357295 26437 raft_consensus.cc:2781] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260925 03:04:50.394888 24543 raft_consensus.cc:3037] T edcf6e6a422d4d198d4fbb0f7d3d845d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:50.335314 26248 consensus_queue.cc:1046] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.045s
I20260925 03:04:50.385519 24466 rpcz_store.cc:276] 0925 03:04:49.189971 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:49.209726 (+ 19755us) service_pool.cc:227] Handling call
0925 03:04:49.209923 (+ 197us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:49.210059 (+ 136us) raft_consensus.cc:1520] Early marking committed up to index 4
0925 03:04:49.442341 (+232282us) log.cc:842] Serialized 14 byte log entry
0925 03:04:50.345443 (+903102us) log.cc:842] Serialized 14 byte log entry
0925 03:04:50.346452 (+ 1009us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:50.346504 (+ 52us) raft_consensus.cc:1631] Marking committed up to 4
0925 03:04:50.346551 (+ 47us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:50.346622 (+ 71us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:50.346731 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":26489,"prepare.queue_time_us":56318,"prepare.run_cpu_time_us":24,"prepare.run_wall_time_us":24,"raft.queue_time_us":91973,"raft.run_cpu_time_us":125,"raft.run_wall_time_us":12425,"spinlock_wait_cycles":16596864,"thread_start_us":1102670,"threads_started":3}
I20260925 03:04:50.357609 26544 leader_election.cc:288] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 6 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:50.397996 24465 raft_consensus.cc:1226] T a7aa47c48404465b94a36fd819fe3c7a P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 4. Current term is 5. Ops: [2.2-4.3]
I20260925 03:04:50.379926 26448 raft_consensus.cc:3037] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Advancing to term 5
W20260925 03:04:50.416774 24436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 236183us
I20260925 03:04:50.301204 26672 raft_consensus.cc:504] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:50.427018 25199 raft_consensus.cc:2454] T b4ebd2b043024a548005c8b6b00b0449 P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 3.
I20260925 03:04:50.384052 26598 leader_election.cc:288] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:50.432992 24619 raft_consensus.cc:2379] T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 6: Already voted for candidate eeab93abfdb141a895002ec9f312296c in this term.
I20260925 03:04:50.384780 26683 raft_consensus.cc:2726] T 8ac466793d094325ace224708b2158ed P 1bc1f2e62ab44af995165bd5116a9d43 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260925 03:04:50.164500 26695 raft_consensus.cc:504] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:50.227743 26706 raft_consensus.cc:504] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:50.391870 25196 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49794 (request call id 213) took 1300 ms. Trace:
I20260925 03:04:50.438951 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367277daf8d04778b3fa4ed94d71580d" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:50.434769 24544 raft_consensus.cc:2454] T 37a157996e944db2906110d87ecb0c54 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 6.
I20260925 03:04:50.363107 26671 raft_consensus.cc:504] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:50.404078 26670 raft_consensus.cc:2726] T 550118bc1a0641ce9803bdbf43f2069f P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260925 03:04:50.443363 24436 leader_election.cc:302] T cc2df40db08649c9a75e90000b878559 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
W20260925 03:04:50.454170 26638 consensus_meta.cc:287] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5: Time spent flushing consensus metadata: real 1.210s user 0.020s sys 0.000s
I20260925 03:04:50.454706 26671 leader_election.cc:288] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:50.362643 26719 raft_consensus.cc:504] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:50.444154 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7691c6b475428c82d6181d693d26f3" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:50.474611 24466 raft_consensus.cc:2454] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 4.
I20260925 03:04:50.448439 24470 raft_consensus.cc:1226] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 4. Current term is 5. Ops: []
I20260925 03:04:50.336162 26679 raft_consensus.cc:504] T b1e85248d3b04b979dd080caeeb02fdb P 1bc1f2e62ab44af995165bd5116a9d43 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:50.483877 24436 leader_election.cc:302] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
W20260925 03:04:50.454588 26621 consensus_meta.cc:287] T 8e5819c15eac4544a57e5ee6c9d4f117 P 869b0a8575b54d08ae2817fad5dd42d5: Time spent flushing consensus metadata: real 0.739s user 0.007s sys 0.005s
W20260925 03:04:50.456498 26619 consensus_meta.cc:287] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5: Time spent flushing consensus metadata: real 1.062s user 0.014s sys 0.005s
W20260925 03:04:50.466593 26700 log.cc:924] Time spent T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.249s user 0.000s sys 0.001s
W20260925 03:04:50.465535 26717 log.cc:924] Time spent T 5278a60f97da4e10aecd50a8d9a9a43c P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.120s user 0.000s sys 0.000s
W20260925 03:04:50.493125 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 93566.6us
W20260925 03:04:50.466418 26608 consensus_meta.cc:287] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c: Time spent flushing consensus metadata: real 0.912s user 0.013s sys 0.004s
I20260925 03:04:50.489864 26598 raft_consensus.cc:2781] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader election won for term 7
I20260925 03:04:50.481273 26644 raft_consensus.cc:482] T a4abe0ea757e4886be4945c5001e529a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:50.465605 26551 leader_election.cc:288] T 04e068a0f80941dc9138cee063f53bab P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 3 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:50.466320 26380 raft_consensus.cc:3032] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [term 6 LEADER]: Stepping down as leader of term 6
I20260925 03:04:50.465682 25191 raft_consensus.cc:1226] T 8ac466793d094325ace224708b2158ed P 1bc1f2e62ab44af995165bd5116a9d43 [term 5 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 4. Current term is 5. Ops: []
I20260925 03:04:50.511941 24586 leader_election.cc:302] T 21c43af5e6844fdaacc8c625fa01fd8d P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:04:50.466106 26639 leader_election.cc:288] T 43366a0a2f9d4344bf4e37ffd290ef8a P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
W20260925 03:04:50.486229 26499 consensus_meta.cc:287] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5: Time spent flushing consensus metadata: real 1.094s user 0.009s sys 0.011s
I20260925 03:04:50.497752 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f68804eb5a244d87bb395d50683c6012" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:50.462185 26386 leader_election.cc:288] T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 6 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:50.465601 26642 leader_election.cc:288] T 293ef69caf514ac98510a68495f54eda P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:50.442239 25196 rpcz_store.cc:276] 0925 03:04:49.091697 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:49.104206 (+ 12509us) service_pool.cc:227] Handling call
0925 03:04:50.385527 (+1281321us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:50.481339 26530 raft_consensus.cc:482] T 0be4e0aa984f4f2b81a58e382e434ad5 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:50.454670 26638 raft_consensus.cc:504] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:50.486611 26637 raft_consensus.cc:2726] T cc2df40db08649c9a75e90000b878559 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260925 03:04:50.503153 24543 raft_consensus.cc:2454] T edcf6e6a422d4d198d4fbb0f7d3d845d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 4.
I20260925 03:04:50.482043 26703 raft_consensus.cc:482] T 550118bc1a0641ce9803bdbf43f2069f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:50.489321 26692 leader_election.cc:288] T 43366a0a2f9d4344bf4e37ffd290ef8a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:50.456574 26695 leader_election.cc:288] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:50.440074 26669 consensus_queue.cc:1046] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.055s
I20260925 03:04:50.492879 26621 raft_consensus.cc:504] T 8e5819c15eac4544a57e5ee6c9d4f117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:50.496759 26619 raft_consensus.cc:504] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:50.634270 25196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "293ef69caf514ac98510a68495f54eda" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43"
I20260925 03:04:50.662643 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:50.663450 24543 raft_consensus.cc:3037] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:04:50.512488 26644 raft_consensus.cc:504] T a4abe0ea757e4886be4945c5001e529a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
W20260925 03:04:50.583526 26709 log.cc:924] Time spent T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5: Append to log took a long time: real 0.289s user 0.000s sys 0.001s
W20260925 03:04:50.624214 26716 log.cc:924] Time spent T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.324s user 0.001s sys 0.000s
I20260925 03:04:50.513267 26380 raft_consensus.cc:729] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [term 6 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:50.709172 24464 raft_consensus.cc:2379] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c in current term 8: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:04:50.707720 24543 raft_consensus.cc:2454] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 7.
I20260925 03:04:50.515130 26598 raft_consensus.cc:686] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 LEADER]: Becoming Leader. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:04:50.640480 26630 leader_election.cc:288] T 4e5332a5d329493991dff994e70ea92c P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:50.523157 26551 raft_consensus.cc:2781] T 21c43af5e6844fdaacc8c625fa01fd8d P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260925 03:04:50.582719 26640 leader_election.cc:288] T a7aa47c48404465b94a36fd819fe3c7a P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
W20260925 03:04:50.720597 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 469565us
I20260925 03:04:50.646121 25190 raft_consensus.cc:1261] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43 [term 8 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260925 03:04:50.624707 26718 raft_consensus.cc:482] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260925 03:04:50.677515 26209 consensus_meta.cc:287] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5: Time spent flushing consensus metadata: real 1.300s user 0.017s sys 0.004s
I20260925 03:04:50.583698 26607 raft_consensus.cc:686] T 48ca6ccd98194cf2af7b5278d15bf4ea P eeab93abfdb141a895002ec9f312296c [term 4 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:50.626981 26448 raft_consensus.cc:504] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:50.510448 26608 raft_consensus.cc:504] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:04:50.720413 26438 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.500s user 0.001s sys 0.017s
I20260925 03:04:50.728912 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e068a0f80941dc9138cee063f53bab" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
W20260925 03:04:50.717388 26724 log.cc:924] Time spent T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.333s user 0.000s sys 0.001s
I20260925 03:04:50.521466 26499 raft_consensus.cc:504] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:50.709259 25197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367277daf8d04778b3fa4ed94d71580d" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:50.667057 26530 raft_consensus.cc:504] T 0be4e0aa984f4f2b81a58e382e434ad5 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:50.723246 26551 raft_consensus.cc:482] T 21c43af5e6844fdaacc8c625fa01fd8d P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting leader election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:50.724799 26450 leader_election.cc:288] T 367277daf8d04778b3fa4ed94d71580d P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 5 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:50.755899 26718 raft_consensus.cc:504] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:50.738479 26648 leader_election.cc:288] T 420163a6cdce412ebc21c25fe3835638 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:50.739138 26640 raft_consensus.cc:482] T c85f5d1d3f004cf5817bb594fd8e6469 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:50.673462 26703 raft_consensus.cc:504] T 550118bc1a0641ce9803bdbf43f2069f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:04:50.757828 26723 log.cc:924] Time spent T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.313s user 0.000s sys 0.001s
I20260925 03:04:50.796730 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b627ecb4a5384ef392aa695e06599c39" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:50.759456 26706 leader_election.cc:288] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:50.758512 25196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e068a0f80941dc9138cee063f53bab" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43"
I20260925 03:04:50.831615 24616 raft_consensus.cc:2379] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 in current term 5: Already voted for candidate eeab93abfdb141a895002ec9f312296c in this term.
I20260925 03:04:50.759025 26209 raft_consensus.cc:504] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:50.758948 26380 consensus_queue.cc:260] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:50.919567 24468 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1069) took 1645 ms. Trace:
I20260925 03:04:51.010977 24543 raft_consensus.cc:3037] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:04:51.014796 24468 rpcz_store.cc:276] 0925 03:04:49.273740 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:49.467694 (+193954us) service_pool.cc:227] Handling call
0925 03:04:49.468041 (+ 347us) raft_consensus.cc:1457] Updating replica for 2 ops
0925 03:04:49.499558 (+ 31517us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:49.499590 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0925 03:04:49.515518 (+ 15928us) log.cc:842] Serialized 52 byte log entry
0925 03:04:49.976459 (+460941us) raft_consensus.cc:1631] Marking committed up to 2
0925 03:04:49.976500 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:49.976571 (+ 71us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:04:50.919268 (+942697us) raft_consensus.cc:1671] finished
0925 03:04:50.919302 (+ 34us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:50.919490 (+ 188us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":37027,"raft.queue_time_us":31817,"raft.run_cpu_time_us":150,"raft.run_wall_time_us":12459,"thread_start_us":424283,"threads_started":1,"wal-append.queue_time_us":668113}
W20260925 03:04:50.961043 26731 log.cc:924] Time spent T a7aa47c48404465b94a36fd819fe3c7a P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.226s user 0.001s sys 0.000s
W20260925 03:04:50.963617 26740 log.cc:924] Time spent T 550118bc1a0641ce9803bdbf43f2069f P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.215s user 0.000s sys 0.000s
W20260925 03:04:50.975056 26687 consensus_meta.cc:287] T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5: Time spent flushing consensus metadata: real 1.173s user 0.008s sys 0.011s
W20260925 03:04:50.954242 26725 log.cc:924] Time spent T 7993f9c3e2a44f408bbf5d1e395e2306 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.427s user 0.000s sys 0.001s
W20260925 03:04:50.957867 26371 consensus_meta.cc:287] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5: Time spent flushing consensus metadata: real 1.162s user 0.011s sys 0.005s
I20260925 03:04:50.833130 26692 raft_consensus.cc:482] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
W20260925 03:04:50.962447 26729 log.cc:924] Time spent T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.216s user 0.000s sys 0.000s
W20260925 03:04:51.007946 26664 consensus_meta.cc:287] T 71f461a6e2ff434993b78a2d077f0abd P eeab93abfdb141a895002ec9f312296c: Time spent flushing consensus metadata: real 1.509s user 0.013s sys 0.005s
W20260925 03:04:50.963137 26751 log.cc:924] Time spent T 06f4a7eaf6534e4986c9fd35911038e4 P 869b0a8575b54d08ae2817fad5dd42d5: Append to log took a long time: real 0.129s user 0.000s sys 0.000s
I20260925 03:04:50.945210 26621 leader_election.cc:288] T 8e5819c15eac4544a57e5ee6c9d4f117 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:50.973109 26644 leader_election.cc:288] T a4abe0ea757e4886be4945c5001e529a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:50.849236 26450 raft_consensus.cc:482] T e6b6ec010caf4689b8c90449764f5cf9 P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:50.951033 26750 consensus_queue.cc:1046] T cc2df40db08649c9a75e90000b878559 P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.317s
I20260925 03:04:50.836315 26719 leader_election.cc:288] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:50.960261 26638 leader_election.cc:288] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 8 election: Requested vote from peers eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:50.951231 26607 consensus_queue.cc:237] T 48ca6ccd98194cf2af7b5278d15bf4ea P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:04:51.144404 26650 consensus_meta.cc:287] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent flushing consensus metadata: real 1.345s user 0.004s sys 0.019s
I20260925 03:04:50.829625 26551 raft_consensus.cc:3037] T 21c43af5e6844fdaacc8c625fa01fd8d P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:50.849808 26640 raft_consensus.cc:504] T c85f5d1d3f004cf5817bb594fd8e6469 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:04:51.237473 26686 consensus_meta.cc:287] T c88cbd39e01f44628a094af516be2833 P 869b0a8575b54d08ae2817fad5dd42d5: Time spent flushing consensus metadata: real 1.433s user 0.017s sys 0.000s
W20260925 03:04:51.359032 26366 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.845s user 0.019s sys 0.031s
I20260925 03:04:51.279258 26664 raft_consensus.cc:2726] T 71f461a6e2ff434993b78a2d077f0abd P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260925 03:04:51.237452 26706 raft_consensus.cc:482] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:51.257275 26448 leader_election.cc:288] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:51.280228 26608 leader_election.cc:288] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:51.259603 26209 leader_election.cc:288] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:51.270097 26499 leader_election.cc:288] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:51.212208 26743 raft_consensus.cc:482] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:51.269660 26530 leader_election.cc:288] T 0be4e0aa984f4f2b81a58e382e434ad5 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:50.928570 26598 consensus_queue.cc:237] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:51.273881 26744 raft_consensus.cc:482] T 5278a60f97da4e10aecd50a8d9a9a43c P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
W20260925 03:04:51.354833 26545 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.549s user 0.014s sys 0.062s
I20260925 03:04:51.278393 26371 raft_consensus.cc:504] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:51.275056 26718 leader_election.cc:288] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
W20260925 03:04:51.366946 26649 consensus_meta.cc:287] T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent flushing consensus metadata: real 1.183s user 0.006s sys 0.007s
I20260925 03:04:51.278872 26692 raft_consensus.cc:504] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:51.236641 26737 raft_consensus.cc:482] T e92f377a79354a6ebb4eec67303c13e3 P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:51.277102 26687 raft_consensus.cc:504] T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:51.281143 26450 raft_consensus.cc:504] T e6b6ec010caf4689b8c90449764f5cf9 P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:51.363713 26686 raft_consensus.cc:504] T c88cbd39e01f44628a094af516be2833 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:51.368429 26706 raft_consensus.cc:504] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:51.380795 26737 raft_consensus.cc:504] T e92f377a79354a6ebb4eec67303c13e3 P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:51.389570 26737 leader_election.cc:288] T e92f377a79354a6ebb4eec67303c13e3 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
W20260925 03:04:51.386411 26690 consensus_meta.cc:287] T 25102ba08e4e49388f6176badcd10a25 P 869b0a8575b54d08ae2817fad5dd42d5: Time spent flushing consensus metadata: real 1.185s user 0.011s sys 0.001s
I20260925 03:04:51.379348 26649 raft_consensus.cc:504] T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
W20260925 03:04:51.392958 26545 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.586s user 0.015s sys 0.062s
I20260925 03:04:51.376359 26744 raft_consensus.cc:504] T 5278a60f97da4e10aecd50a8d9a9a43c P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:51.399809 26687 leader_election.cc:288] T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:51.400866 26692 leader_election.cc:288] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:51.401870 26706 leader_election.cc:288] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:51.397524 26690 raft_consensus.cc:504] T 25102ba08e4e49388f6176badcd10a25 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:51.406044 26686 leader_election.cc:288] T c88cbd39e01f44628a094af516be2833 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:51.377365 26743 raft_consensus.cc:504] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:51.406750 26640 leader_election.cc:288] T c85f5d1d3f004cf5817bb594fd8e6469 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:51.408214 26649 leader_election.cc:288] T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 election: Requested vote from peers eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:51.406790 26690 leader_election.cc:288] T 25102ba08e4e49388f6176badcd10a25 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:51.391350 26679 leader_election.cc:288] T b1e85248d3b04b979dd080caeeb02fdb P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:51.408301 26743 leader_election.cc:288] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:51.410301 26744 leader_election.cc:288] T 5278a60f97da4e10aecd50a8d9a9a43c P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:51.444923 24472 raft_consensus.cc:1226] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 5. Current term is 6. Ops: []
W20260925 03:04:51.447024 24514 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 1.071s user 0.001s sys 0.042s
W20260925 03:04:51.447628 24514 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.071s user 0.001s sys 0.042s
W20260925 03:04:51.448197 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 1.07557e+06us
I20260925 03:04:50.837028 24467 raft_consensus.cc:1226] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 5. Current term is 6. Ops: []
W20260925 03:04:51.454592 24514 connection.cc:232] RPC call timeout handler was delayed by -1.0111217700014095s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1807s for a response
I20260925 03:04:51.454931 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7691c6b475428c82d6181d693d26f3" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:51.457533 24471 raft_consensus.cc:1226] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20260925 03:04:51.459234 24473 raft_consensus.cc:2379] T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c in current term 4: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:04:51.459345 24539 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 919) took 1936 ms. Trace:
I20260925 03:04:51.460273 24468 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1277) took 1213 ms. Trace:
I20260925 03:04:51.460783 24468 rpcz_store.cc:276] 0925 03:04:50.246741 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:51.454042 (+1207301us) service_pool.cc:227] Handling call
0925 03:04:51.460139 (+ 6097us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:51.461127 24540 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 914) took 1964 ms. Trace:
I20260925 03:04:51.464313 24540 rpcz_store.cc:276] 0925 03:04:49.496163 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:49.555596 (+ 59433us) service_pool.cc:227] Handling call
0925 03:04:49.936660 (+381064us) spinlock_profiling.cc:243] Waited 3.87 ms on lock 0x7b5800003c98. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f43733ecf11 00007f4373570851 00007f4373570702 00007f437c460b09 00007f437b39c730 00007f437b46279b 00007f437b462821 00007f437b4262b0 00007f437b4117f9 00007f437b4112a2 00007f4374af7684
0925 03:04:49.936902 (+ 242us) raft_consensus.cc:1457] Updating replica for 2 ops
0925 03:04:49.957220 (+ 20318us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:49.957251 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0925 03:04:49.966363 (+ 9112us) log.cc:842] Serialized 52 byte log entry
0925 03:04:50.398778 (+432415us) raft_consensus.cc:1631] Marking committed up to 1
0925 03:04:50.398814 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:50.399509 (+ 695us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:04:51.460486 (+1060977us) raft_consensus.cc:1671] finished
0925 03:04:51.460535 (+ 49us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:51.460960 (+ 425us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":135802,"raft.run_cpu_time_us":206,"raft.run_wall_time_us":49556,"spinlock_wait_cycles":8894848,"thread_start_us":384967,"threads_started":1,"wal-append.queue_time_us":653034}
I20260925 03:04:51.467449 24542 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 923) took 1942 ms. Trace:
I20260925 03:04:51.467864 24545 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33068 (request call id 132) took 2006 ms. Trace:
I20260925 03:04:51.469466 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e725b82985746b39c0cc8f671c95353" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:51.468331 24545 rpcz_store.cc:276] 0925 03:04:49.460969 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:49.461747 (+ 778us) service_pool.cc:227] Handling call
0925 03:04:49.462068 (+ 321us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:04:49.462421 (+ 353us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:49.462450 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:04:49.466861 (+ 4411us) log.cc:842] Serialized 26 byte log entry
0925 03:04:49.761564 (+294703us) spinlock_profiling.cc:243] Waited 2.08 ms on lock 0x7b18003e4520. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f437c4478b1 00007f437c454661 00007f437358a345 00007f437c3c1a39 00007f43735a78bb 00007f43735a5cbf 00007f43735a7e22 00007f43761515bb 00007f4376157063 00007f4376157411 00007f43764a45cf
0925 03:04:49.971363 (+209799us) raft_consensus.cc:1631] Marking committed up to 7
0925 03:04:50.374554 (+403191us) log.cc:842] Serialized 14 byte log entry
0925 03:04:50.374847 (+ 293us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:50.374958 (+ 111us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:04:51.400641 (+1025683us) spinlock_profiling.cc:243] Waited 4.23 ms on lock 0x7b18003e4520. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f437c4478b1 00007f437c454661 00007f43735a8638 00007f43735abb11 00007f43735abac9 00007f43735aba59 00007f43735aba21 00007f43735aad1d 00007f437c80c204 00007f437c80c039 00007f437358adc6
0925 03:04:51.467360 (+ 66719us) raft_consensus.cc:1671] finished
0925 03:04:51.467395 (+ 35us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:51.467679 (+ 284us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":207182,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":42,"raft.queue_time_us":6567,"raft.run_cpu_time_us":248,"raft.run_wall_time_us":36189,"spinlock_wait_cycles":14503168,"thread_start_us":562858,"threads_started":2,"wal-append.queue_time_us":749898}
I20260925 03:04:51.470412 24540 raft_consensus.cc:3037] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:04:51.471885 24545 raft_consensus.cc:1226] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 6. Current term is 7. Ops: [6.5-6.5]
I20260925 03:04:51.473670 24544 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 1414) took 1546 ms. Trace:
I20260925 03:04:51.474316 24544 rpcz_store.cc:276] 0925 03:04:49.927177 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:50.519692 (+592515us) service_pool.cc:227] Handling call
0925 03:04:50.575728 (+ 56036us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:04:50.575995 (+ 267us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:50.576021 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:04:50.705116 (+129095us) log.cc:842] Serialized 26 byte log entry
0925 03:04:50.758563 (+ 53447us) raft_consensus.cc:1631] Marking committed up to 4
0925 03:04:50.769732 (+ 11169us) log.cc:842] Serialized 14 byte log entry
0925 03:04:50.769991 (+ 259us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:50.770065 (+ 74us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:04:51.473215 (+703150us) raft_consensus.cc:1671] finished
0925 03:04:51.473250 (+ 35us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:51.473493 (+ 243us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":11916,"prepare.run_wall_time_us":52,"raft.queue_time_us":851,"spinlock_wait_cycles":218655232,"thread_start_us":23572,"threads_started":1,"wal-append.queue_time_us":645052}
I20260925 03:04:51.473661 24547 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 915) took 1977 ms. Trace:
I20260925 03:04:51.473608 24546 raft_consensus.cc:2379] T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1bc1f2e62ab44af995165bd5116a9d43 in current term 4: Already voted for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in this term.
I20260925 03:04:51.476931 24544 raft_consensus.cc:1226] T 37a157996e944db2906110d87ecb0c54 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 5. Current term is 6. Ops: []
I20260925 03:04:51.477276 24545 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 1415) took 1545 ms. Trace:
I20260925 03:04:51.480121 24545 rpcz_store.cc:276] 0925 03:04:49.927504 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:51.470796 (+1543292us) service_pool.cc:227] Handling call
0925 03:04:51.471460 (+ 664us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:04:51.472379 (+ 919us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:51.472603 (+ 224us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260925 03:04:51.483284 24585 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.640s user 0.002s sys 0.002s
W20260925 03:04:51.483711 24585 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.641s user 0.003s sys 0.002s
I20260925 03:04:51.477309 24547 rpcz_store.cc:276] 0925 03:04:49.496365 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:49.551581 (+ 55216us) service_pool.cc:227] Handling call
0925 03:04:49.620472 (+ 68891us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:04:49.971864 (+351392us) raft_consensus.cc:1520] Early marking committed up to index 2
0925 03:04:50.341800 (+369936us) log.cc:842] Serialized 14 byte log entry
0925 03:04:50.439708 (+ 97908us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:04:50.445799 (+ 6091us) log.cc:842] Serialized 26 byte log entry
0925 03:04:50.446048 (+ 249us) raft_consensus.cc:1631] Marking committed up to 3
0925 03:04:50.447315 (+ 1267us) log.cc:842] Serialized 14 byte log entry
0925 03:04:50.477741 (+ 30426us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:50.477818 (+ 77us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:04:51.473122 (+995304us) raft_consensus.cc:1671] finished
0925 03:04:51.473170 (+ 48us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:51.473483 (+ 313us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":57781,"prepare.queue_time_us":243092,"prepare.run_cpu_time_us":70,"prepare.run_wall_time_us":72,"raft.queue_time_us":200455,"raft.run_cpu_time_us":295,"raft.run_wall_time_us":23663,"spinlock_wait_cycles":44524672,"thread_start_us":658062,"threads_started":3,"wal-append.queue_time_us":330059}
I20260925 03:04:51.485889 26644 raft_consensus.cc:482] T 48ca6ccd98194cf2af7b5278d15bf4ea P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:51.486742 26438 raft_consensus.cc:482] T 100c113a9fd84cc8adf2ec7b33ea0093 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:51.488448 26650 raft_consensus.cc:482] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:51.482385 24544 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 927) took 1540 ms. Trace:
W20260925 03:04:51.359869 26366 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.853s user 0.019s sys 0.031s
I20260925 03:04:51.528517 24616 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38570 (request call id 1466) took 1569 ms. Trace:
I20260925 03:04:51.486960 26644 raft_consensus.cc:504] T 48ca6ccd98194cf2af7b5278d15bf4ea P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:51.487633 26649 raft_consensus.cc:482] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:51.493347 26692 raft_consensus.cc:482] T 8e5819c15eac4544a57e5ee6c9d4f117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:51.494166 26706 raft_consensus.cc:482] T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:51.529063 24616 rpcz_store.cc:276] 0925 03:04:49.958915 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:49.961720 (+ 2805us) service_pool.cc:227] Handling call
0925 03:04:51.528070 (+1566350us) spinlock_profiling.cc:243] Waited 824 ms on lock 0x7b540018f1d4. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d42ae 00007f437b411c27 00007f4374af9c04 00007f4374af9b92 00007f4374af9ac1 00007f4374af9a3c 00007f4374af8cf2 00007f43748fa94c 00007f43748f9d86 00007f43748f972f
0925 03:04:51.528313 (+ 243us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1895735680}
I20260925 03:04:51.497831 26450 leader_election.cc:288] T e6b6ec010caf4689b8c90449764f5cf9 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:51.487929 26438 raft_consensus.cc:504] T 100c113a9fd84cc8adf2ec7b33ea0093 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:51.498991 26686 raft_consensus.cc:482] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260925 03:04:51.499476 24586 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.539s user 0.009s sys 0.039s
I20260925 03:04:51.502547 26371 leader_election.cc:288] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:51.504217 24613 raft_consensus.cc:1226] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 6. Current term is 7. Ops: []
I20260925 03:04:51.467908 24542 rpcz_store.cc:276] 0925 03:04:49.525014 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:49.697818 (+172804us) service_pool.cc:227] Handling call
0925 03:04:49.698121 (+ 303us) raft_consensus.cc:1457] Updating replica for 2 ops
0925 03:04:50.007038 (+308917us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:50.007071 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0925 03:04:50.012570 (+ 5499us) log.cc:842] Serialized 52 byte log entry
0925 03:04:50.343678 (+331108us) raft_consensus.cc:1631] Marking committed up to 1
0925 03:04:50.343713 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:50.343778 (+ 65us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:04:51.466792 (+1123014us) raft_consensus.cc:1671] finished
0925 03:04:51.466834 (+ 42us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:51.467247 (+ 413us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":105053,"raft.run_cpu_time_us":98,"raft.run_wall_time_us":97,"thread_start_us":518158,"threads_started":2,"wal-append.queue_time_us":419084}
I20260925 03:04:51.489408 26650 raft_consensus.cc:504] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:51.508013 25196 raft_consensus.cc:2454] T 04e068a0f80941dc9138cee063f53bab P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 3.
I20260925 03:04:51.459837 24539 rpcz_store.cc:276] 0925 03:04:49.522251 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:49.565185 (+ 42934us) service_pool.cc:227] Handling call
0925 03:04:49.620520 (+ 55335us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:04:50.144240 (+523720us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:50.144277 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:04:50.144866 (+ 589us) log.cc:842] Serialized 26 byte log entry
0925 03:04:50.311318 (+166452us) raft_consensus.cc:1631] Marking committed up to 2
0925 03:04:50.311359 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:50.311440 (+ 81us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:04:51.458664 (+1147224us) raft_consensus.cc:1671] finished
0925 03:04:51.458716 (+ 52us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:51.459101 (+ 385us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":169201,"raft.run_cpu_time_us":56,"raft.run_wall_time_us":57,"spinlock_wait_cycles":29312,"thread_start_us":581077,"threads_started":2,"wal-append.queue_time_us":482154}
I20260925 03:04:51.509266 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1414dcfe53f64a78ae413531eb8ec117" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:51.510418 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37a157996e944db2906110d87ecb0c54" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
W20260925 03:04:51.526780 24615 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34546 (request call id 131) took 1533 ms (client timeout 1935 ms). Trace:
I20260925 03:04:51.491145 24544 rpcz_store.cc:276] 0925 03:04:49.939144 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:51.476088 (+1536944us) service_pool.cc:227] Handling call
0925 03:04:51.476606 (+ 518us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:51.479334 (+ 2728us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:51.479551 (+ 217us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:51.513942 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b627ecb4a5384ef392aa695e06599c39" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:51.494879 26718 raft_consensus.cc:482] T c88cbd39e01f44628a094af516be2833 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:51.535705 26706 raft_consensus.cc:504] T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:51.547050 26650 leader_election.cc:288] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:51.547941 24620 raft_consensus.cc:1261] T b1e85248d3b04b979dd080caeeb02fdb P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260925 03:04:51.536891 26692 raft_consensus.cc:504] T 8e5819c15eac4544a57e5ee6c9d4f117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:51.548130 26650 raft_consensus.cc:482] T c85f5d1d3f004cf5817bb594fd8e6469 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:51.549062 26650 raft_consensus.cc:504] T c85f5d1d3f004cf5817bb594fd8e6469 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:51.549551 24614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9357d144aa49789faaf31fe5e6bf89" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:51.550227 26692 leader_election.cc:288] T 8e5819c15eac4544a57e5ee6c9d4f117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:51.537752 26438 leader_election.cc:288] T 100c113a9fd84cc8adf2ec7b33ea0093 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:51.538683 26764 raft_consensus.cc:482] T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:51.550917 25191 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48424 (request call id 217) took 1657 ms. Trace:
I20260925 03:04:51.551286 26650 leader_election.cc:288] T c85f5d1d3f004cf5817bb594fd8e6469 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:51.551486 26764 raft_consensus.cc:504] T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:51.554447 26764 leader_election.cc:288] T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
W20260925 03:04:51.541421 24586 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.598s user 0.010s sys 0.039s
I20260925 03:04:51.524497 26719 raft_consensus.cc:482] T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:51.554725 25192 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48408 (request call id 188) took 1565 ms. Trace:
I20260925 03:04:51.555141 26686 raft_consensus.cc:504] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:04:51.555713 25194 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48424 (request call id 215) took 2130 ms (client timeout 1556 ms). Trace:
I20260925 03:04:51.556146 24618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367277daf8d04778b3fa4ed94d71580d" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:51.557797 26686 leader_election.cc:288] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:51.535636 26649 raft_consensus.cc:504] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:51.559952 25198 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49794 (request call id 217) took 1520 ms. Trace:
W20260925 03:04:51.560729 24619 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38564 (request call id 1316) took 2352 ms (client timeout 1610 ms). Trace:
I20260925 03:04:51.545400 26718 raft_consensus.cc:504] T c88cbd39e01f44628a094af516be2833 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:51.560364 25198 rpcz_store.cc:276] 0925 03:04:50.039066 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:50.039770 (+ 704us) service_pool.cc:227] Handling call
0925 03:04:51.558611 (+1518841us) spinlock_profiling.cc:243] Waited 35.5 ms on lock 0x7b5800004e98. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f43733ecf11 00007f4373570851 00007f4373570702 00007f437c460b09 00007f437b39c730 00007f437b46279b 00007f437b462821 00007f437b4262b0 00007f437b4117f9 00007f437b4112a2 00007f4374af7684
0925 03:04:51.559031 (+ 420us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:51.559428 (+ 397us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:51.559450 (+ 22us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:51.559516 (+ 66us) raft_consensus.cc:1631] Marking committed up to 6
0925 03:04:51.559587 (+ 71us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:51.559655 (+ 68us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:51.559782 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":81639168}
W20260925 03:04:51.543522 24615 rpcz_store.cc:269] 0925 03:04:49.992937 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:49.993149 (+ 212us) service_pool.cc:227] Handling call
0925 03:04:51.526340 (+1533191us) spinlock_profiling.cc:243] Waited 230 ms on lock 0x7b54000a4e54. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d42ae 00007f437b411c27 00007f4374af9c04 00007f4374af9b92 00007f4374af9ac1 00007f4374af9a3c 00007f4374af8cf2 00007f43748fa94c 00007f43748f9d86 00007f43748f972f
0925 03:04:51.526558 (+ 218us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":528872832}
W20260925 03:04:51.561173 24619 rpcz_store.cc:269] 0925 03:04:49.208523 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:49.208645 (+ 122us) service_pool.cc:227] Handling call
0925 03:04:51.560266 (+2351621us) spinlock_profiling.cc:243] Waited 1.19 s on lock 0x7b54000735d4. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d42ae 00007f437b411c27 00007f4374af9c04 00007f4374af9b92 00007f4374af9ac1 00007f4374af9a3c 00007f4374af8cf2 00007f43748fa94c 00007f43748f9d86 00007f43748f972f
0925 03:04:51.560509 (+ 243us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2746556672}
I20260925 03:04:51.562590 25193 raft_consensus.cc:1226] T 367277daf8d04778b3fa4ed94d71580d P 1bc1f2e62ab44af995165bd5116a9d43 [term 5 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 4. Current term is 5. Ops: []
I20260925 03:04:51.555475 26719 raft_consensus.cc:504] T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:51.551438 25191 rpcz_store.cc:276] 0925 03:04:49.893551 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:49.908780 (+ 15229us) service_pool.cc:227] Handling call
0925 03:04:50.384772 (+475992us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:50.516927 (+132155us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:51.549318 (+1032391us) spinlock_profiling.cc:243] Waited 51.2 ms on lock 0x7b540043ce54. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d0f51 00007f43764cf99e 00007f437b4112e9 00007f4374af7684 00007f4374af7612 00007f4374af7541 00007f4374af74bc 00007f4374af6772 00007f43748fa94c 00007f43748f9d86
0925 03:04:51.550459 (+ 1141us) spinlock_profiling.cc:243] Waited 449 ms on lock 0x7b540043ce50. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764cfa09 00007f437b4112e9 00007f4374af7684 00007f4374af7612 00007f4374af7541 00007f4374af74bc 00007f4374af6772 00007f43748fa94c 00007f43748f9d86 00007f43748f972f
0925 03:04:51.550679 (+ 220us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1150259712}
I20260925 03:04:51.567960 26718 leader_election.cc:288] T c88cbd39e01f44628a094af516be2833 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:51.568876 26644 leader_election.cc:288] T 48ca6ccd98194cf2af7b5278d15bf4ea P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:51.560981 26649 leader_election.cc:288] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
W20260925 03:04:51.556216 25194 rpcz_store.cc:269] 0925 03:04:49.425655 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:49.429330 (+ 3675us) service_pool.cc:227] Handling call
0925 03:04:51.555267 (+2125937us) spinlock_profiling.cc:243] Waited 841 ms on lock 0x7b54005bf754. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d42ae 00007f437b411c27 00007f4374af9c04 00007f4374af9b92 00007f4374af9ac1 00007f4374af9a3c 00007f4374af8cf2 00007f43748fa94c 00007f43748f9d86 00007f43748f972f
0925 03:04:51.555497 (+ 230us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1933582592}
I20260925 03:04:51.570611 24541 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 918) took 2022 ms. Trace:
I20260925 03:04:51.548175 26706 leader_election.cc:288] T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:51.571872 24539 raft_consensus.cc:1226] T f22ddd7d1c444dc58f45a1fb0b3e02bc P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 5. Current term is 6. Ops: []
I20260925 03:04:51.572202 26247 raft_consensus.cc:482] T 442d4a29b1d743599f4b6b4871e26668 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:51.573050 26247 raft_consensus.cc:504] T 442d4a29b1d743599f4b6b4871e26668 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:51.574668 24614 raft_consensus.cc:1226] T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 5. Current term is 6. Ops: []
I20260925 03:04:51.575464 26247 leader_election.cc:288] T 442d4a29b1d743599f4b6b4871e26668 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:51.575964 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43366a0a2f9d4344bf4e37ffd290ef8a" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:51.577123 24616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c81a1b3b616492ab6c964f45af36077" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:51.573009 24539 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 931) took 1053 ms. Trace:
I20260925 03:04:51.583947 24539 rpcz_store.cc:276] 0925 03:04:50.519783 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:51.570161 (+1050378us) service_pool.cc:227] Handling call
0925 03:04:51.570738 (+ 577us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:51.572618 (+ 1880us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:51.572889 (+ 271us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:51.585714 26450 raft_consensus.cc:482] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:51.585909 24616 raft_consensus.cc:2454] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 4.
I20260925 03:04:51.586526 26450 raft_consensus.cc:504] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:51.591425 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e725b82985746b39c0cc8f671c95353" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:51.580644 24585 leader_election.cc:302] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, eeab93abfdb141a895002ec9f312296c; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:51.865386 24540 raft_consensus.cc:2454] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 6.
W20260925 03:04:51.916216 24464 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 1064) took 2707 ms (client timeout 1971 ms). Trace:
I20260925 03:04:51.976876 24471 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1068) took 2703 ms. Trace:
I20260925 03:04:51.640468 24619 raft_consensus.cc:2354] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 5. Current term is 6.
I20260925 03:04:51.571270 24541 rpcz_store.cc:276] 0925 03:04:49.521881 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:49.562937 (+ 41056us) service_pool.cc:227] Handling call
0925 03:04:49.563389 (+ 452us) raft_consensus.cc:1457] Updating replica for 2 ops
0925 03:04:49.564679 (+ 1290us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:49.564751 (+ 72us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0925 03:04:49.565314 (+ 563us) log.cc:842] Serialized 52 byte log entry
0925 03:04:49.922375 (+357061us) raft_consensus.cc:1631] Marking committed up to 6
0925 03:04:49.961620 (+ 39245us) log.cc:842] Serialized 14 byte log entry
0925 03:04:49.961929 (+ 309us) log.cc:842] Serialized 14 byte log entry
0925 03:04:51.523884 (+1561955us) spinlock_profiling.cc:243] Waited 12 ms on lock 0x7b1800280360. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f437c4478b1 00007f437c454661 00007f437615f82c 00007f437358a9dc 00007f437c3c1a39 00007f43735a78bb 00007f43735a5cbf 00007f43735a5155 00007f43764c83d0 00007f43764da576 00007f43764eb0d7
0925 03:04:51.524199 (+ 315us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:51.531339 (+ 7140us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:04:51.539755 (+ 8416us) raft_consensus.cc:1671] finished
0925 03:04:51.539781 (+ 26us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:51.544591 (+ 4810us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":14098,"prepare.queue_time_us":469924,"prepare.run_cpu_time_us":73,"prepare.run_wall_time_us":15840,"raft.queue_time_us":600173,"raft.run_cpu_time_us":284,"raft.run_wall_time_us":22142,"spinlock_wait_cycles":36162304,"thread_start_us":448851,"threads_started":2,"wal-append.queue_time_us":502903}
W20260925 03:04:51.961246 24469 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 1046) took 2736 ms (client timeout 1692 ms). Trace:
I20260925 03:04:51.974908 24467 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1077) took 2357 ms. Trace:
W20260925 03:04:51.878847 24439 service_pool.cc:137] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:55562 dropped due to backpressure. The service queue is full; it has 50 items. [suppressed 2 similar messages]
W20260925 03:04:51.605532 24468 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1288) took 1306 ms (client timeout 1610 ms). Trace:
I20260925 03:04:51.640491 24621 raft_consensus.cc:1226] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 4. Current term is 5. Ops: []
W20260925 03:04:51.884863 24543 consensus_meta.cc:287] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent flushing consensus metadata: real 0.849s user 0.006s sys 0.007s
W20260925 03:04:52.246630 24464 rpcz_store.cc:269] 0925 03:04:49.208173 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:49.209072 (+ 899us) service_pool.cc:227] Handling call
0925 03:04:51.915726 (+2706654us) spinlock_profiling.cc:243] Waited 1.46 s on lock 0x7b540019f854. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d42ae 00007f437b411c27 00007f4374af9c04 00007f4374af9b92 00007f4374af9ac1 00007f4374af9a3c 00007f4374af8cf2 00007f43748fa94c 00007f43748f9d86 00007f43748f972f
0925 03:04:51.915988 (+ 262us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3365042944}
W20260925 03:04:52.240108 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 289656us
I20260925 03:04:51.782931 24545 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 928) took 1840 ms. Trace:
I20260925 03:04:51.718796 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f68804eb5a244d87bb395d50683c6012" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
W20260925 03:04:51.640393 24436 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.907s user 0.004s sys 0.004s
W20260925 03:04:51.976156 24473 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 1075) took 2435 ms (client timeout 2239 ms). Trace:
W20260925 03:04:52.280171 24468 rpcz_store.cc:269] 0925 03:04:50.279157 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:51.513511 (+1234354us) service_pool.cc:227] Handling call
0925 03:04:51.585091 (+ 71580us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260925 03:04:52.293371 24540 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36786 (request call id 942) took 1668 ms (client timeout 2140 ms). Trace:
W20260925 03:04:52.294054 24540 rpcz_store.cc:269] 0925 03:04:50.583474 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:51.468833 (+885359us) service_pool.cc:227] Handling call
0925 03:04:52.251440 (+782607us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:52.294191 24545 rpcz_store.cc:276] 0925 03:04:49.939432 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:51.529972 (+1590540us) service_pool.cc:227] Handling call
0925 03:04:51.605981 (+ 76009us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:51.719536 (+113555us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:51.719566 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:51.779865 (+ 60299us) raft_consensus.cc:1631] Marking committed up to 3
0925 03:04:51.779919 (+ 54us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:51.779997 (+ 78us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:51.780140 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:52.247737 24471 rpcz_store.cc:276] 0925 03:04:49.273397 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:49.388383 (+114986us) service_pool.cc:227] Handling call
0925 03:04:49.388733 (+ 350us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:04:51.458204 (+2069471us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:51.976528 (+518324us) spinlock_profiling.cc:243] Waited 2.07 s on lock 0x7b54001ab3d4. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d0f51 00007f43764cf99e 00007f437b4112e9 00007f4374af7684 00007f4374af7612 00007f4374af7541 00007f4374af74bc 00007f4374af6772 00007f43748fa94c 00007f43748f9d86
0925 03:04:51.976798 (+ 270us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4756032768}
I20260925 03:04:52.251773 24467 rpcz_store.cc:276] 0925 03:04:49.617280 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:49.918854 (+301574us) service_pool.cc:227] Handling call
0925 03:04:49.936331 (+ 17477us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:51.453550 (+1517219us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:51.974367 (+520817us) spinlock_profiling.cc:243] Waited 900 ms on lock 0x7b54001abdd4. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d0f51 00007f43764cf99e 00007f437b4112e9 00007f4374af7684 00007f4374af7612 00007f4374af7541 00007f4374af74bc 00007f4374af6772 00007f43748fa94c 00007f43748f9d86
0925 03:04:51.974701 (+ 334us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2070999936}
W20260925 03:04:51.880740 26551 consensus_meta.cc:287] T 21c43af5e6844fdaacc8c625fa01fd8d P eeab93abfdb141a895002ec9f312296c: Time spent flushing consensus metadata: real 0.525s user 0.000s sys 0.015s
W20260925 03:04:52.251351 24469 rpcz_store.cc:269] 0925 03:04:49.181056 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:49.181178 (+ 122us) service_pool.cc:227] Handling call
0925 03:04:51.917366 (+2736188us) spinlock_profiling.cc:243] Waited 946 ms on lock 0x7b540040d3d4. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d42ae 00007f437b411c27 00007f4374af9c04 00007f4374af9b92 00007f4374af9ac1 00007f4374af9a3c 00007f4374af8cf2 00007f43748fa94c 00007f43748f9d86 00007f43748f972f
0925 03:04:51.917540 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2175094528}
W20260925 03:04:52.298462 24436 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.673s user 0.007s sys 0.025s
I20260925 03:04:51.634070 26450 leader_election.cc:288] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
W20260925 03:04:52.293356 24473 rpcz_store.cc:269] 0925 03:04:49.540988 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:49.769360 (+228372us) service_pool.cc:227] Handling call
0925 03:04:51.975843 (+2206483us) spinlock_profiling.cc:243] Waited 1.68 s on lock 0x7b540019f354. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d42ae 00007f437b411c27 00007f4374af9c04 00007f4374af9b92 00007f4374af9ac1 00007f4374af9a3c 00007f4374af8cf2 00007f43748fa94c 00007f43748f9d86 00007f43748f972f
0925 03:04:51.976057 (+ 214us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3875466112}
I20260925 03:04:52.251610 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367277daf8d04778b3fa4ed94d71580d" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:51.714632 26672 leader_election.cc:288] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:51.699865 26679 raft_consensus.cc:482] T 71f461a6e2ff434993b78a2d077f0abd P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:52.034806 26619 leader_election.cc:288] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:52.307960 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be4e0aa984f4f2b81a58e382e434ad5" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:52.308982 24548 raft_consensus.cc:2454] T 0be4e0aa984f4f2b81a58e382e434ad5 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 4.
I20260925 03:04:52.309064 26701 raft_consensus.cc:482] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:52.310011 26703 leader_election.cc:288] T 550118bc1a0641ce9803bdbf43f2069f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:52.309756 26701 raft_consensus.cc:504] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:52.310240 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e92f377a79354a6ebb4eec67303c13e3" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:52.311439 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5278a60f97da4e10aecd50a8d9a9a43c" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:52.312107 26701 leader_election.cc:288] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:52.312284 24541 raft_consensus.cc:2454] T 5278a60f97da4e10aecd50a8d9a9a43c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 3.
I20260925 03:04:52.313398 26699 raft_consensus.cc:3032] T 37a157996e944db2906110d87ecb0c54 P eeab93abfdb141a895002ec9f312296c [term 5 LEADER]: Stepping down as leader of term 5
I20260925 03:04:52.313802 26699 raft_consensus.cc:729] T 37a157996e944db2906110d87ecb0c54 P eeab93abfdb141a895002ec9f312296c [term 5 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:52.303697 26551 raft_consensus.cc:504] T 21c43af5e6844fdaacc8c625fa01fd8d P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:52.314968 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367277daf8d04778b3fa4ed94d71580d" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:52.315143 26699 consensus_queue.cc:260] T 37a157996e944db2906110d87ecb0c54 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:52.315871 24540 raft_consensus.cc:3037] T 367277daf8d04778b3fa4ed94d71580d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:51.885232 26756 raft_consensus.cc:3032] T cc2df40db08649c9a75e90000b878559 P eeab93abfdb141a895002ec9f312296c [term 4 LEADER]: Stepping down as leader of term 4
I20260925 03:04:52.316514 26756 raft_consensus.cc:729] T cc2df40db08649c9a75e90000b878559 P eeab93abfdb141a895002ec9f312296c [term 4 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
W20260925 03:04:52.319231 24464 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1300) took 1972 ms (client timeout 1548 ms). Trace:
I20260925 03:04:52.320932 24548 raft_consensus.cc:2454] T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 4.
W20260925 03:04:52.320560 24464 rpcz_store.cc:269] 0925 03:04:50.346536 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:52.318525 (+1971989us) service_pool.cc:214] Skipping call since client already timed out
0925 03:04:52.318949 (+ 424us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260925 03:04:51.878774 26720 raft_consensus.cc:3032] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 LEADER]: Stepping down as leader of term 6
I20260925 03:04:52.313925 24541 raft_consensus.cc:1261] T 25102ba08e4e49388f6176badcd10a25 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260925 03:04:51.563750 25192 rpcz_store.cc:276] 0925 03:04:49.989336 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:50.012897 (+ 23561us) service_pool.cc:227] Handling call
0925 03:04:51.553428 (+1540531us) spinlock_profiling.cc:243] Waited 51.1 ms on lock 0x7b5800004e98. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f43733ecf11 00007f4373570851 00007f4373570bce 000000000034aea9 00007f437b39ca30 00007f437b4627b4 00007f437b462821 00007f437b4262b0 00007f437b4117f9 00007f437b4112a2 00007f4374af7684
0925 03:04:51.553728 (+ 300us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:51.554179 (+ 451us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:51.554208 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:51.554263 (+ 55us) raft_consensus.cc:1631] Marking committed up to 6
0925 03:04:51.554298 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:51.554383 (+ 85us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:51.554521 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":117577984}
I20260925 03:04:51.595651 26764 raft_consensus.cc:482] T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:52.327785 24540 raft_consensus.cc:2454] T 367277daf8d04778b3fa4ed94d71580d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 5.
I20260925 03:04:52.328446 26508 raft_consensus.cc:3032] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [term 4 LEADER]: Stepping down as leader of term 4
I20260925 03:04:52.333179 24541 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 932) took 1810 ms. Trace:
I20260925 03:04:51.713476 26737 raft_consensus.cc:482] T 29637f5a4a504ce6ba29033ccd0cd716 P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:52.333552 24541 rpcz_store.cc:276] 0925 03:04:50.520060 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:52.313075 (+1793015us) service_pool.cc:227] Handling call
0925 03:04:52.313499 (+ 424us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:52.329909 (+ 16410us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:52.330101 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:51.835777 26766 raft_consensus.cc:482] T ceab6b0fccd54e909a64f66fe81a9a1f P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:52.331218 26378 raft_consensus.cc:3032] T edcf6e6a422d4d198d4fbb0f7d3d845d P eeab93abfdb141a895002ec9f312296c [term 3 LEADER]: Stepping down as leader of term 3
I20260925 03:04:51.793464 26682 raft_consensus.cc:482] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:52.316536 26699 raft_consensus.cc:3037] T 37a157996e944db2906110d87ecb0c54 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:04:52.337188 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e8af2641e124f9589c5ccfdac785f63" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:52.324827 24543 raft_consensus.cc:2454] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 6.
I20260925 03:04:52.335013 26450 raft_consensus.cc:482] T 7e49248643df4131a1c42e74244e05c2 P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:52.333010 26508 raft_consensus.cc:729] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [term 4 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:52.330189 26720 raft_consensus.cc:729] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
W20260925 03:04:52.347973 24464 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1282) took 2094 ms (client timeout 1696 ms). Trace:
W20260925 03:04:52.351845 24464 rpcz_store.cc:269] 0925 03:04:50.250212 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:52.339750 (+2089538us) service_pool.cc:214] Skipping call since client already timed out
0925 03:04:52.344348 (+ 4598us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260925 03:04:52.337442 26378 raft_consensus.cc:729] T edcf6e6a422d4d198d4fbb0f7d3d845d P eeab93abfdb141a895002ec9f312296c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:52.342633 26682 raft_consensus.cc:3037] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:04:52.356539 24545 raft_consensus.cc:3037] T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Advancing to term 5
W20260925 03:04:52.357957 24543 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36786 (request call id 943) took 1726 ms (client timeout 2015 ms). Trace:
W20260925 03:04:52.359316 24543 rpcz_store.cc:269] 0925 03:04:50.628961 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:50.792771 (+163810us) service_pool.cc:227] Handling call
0925 03:04:52.351056 (+1558285us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:52.362571 26741 raft_consensus.cc:2781] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Leader election won for term 5
W20260925 03:04:52.363679 24546 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33068 (request call id 133) took 2227 ms (client timeout 1935 ms). Trace:
I20260925 03:04:52.363333 26508 consensus_queue.cc:260] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:52.365159 24539 raft_consensus.cc:1226] T 367277daf8d04778b3fa4ed94d71580d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 4. Current term is 5. Ops: []
W20260925 03:04:52.364136 24546 rpcz_store.cc:269] 0925 03:04:50.136053 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:50.251577 (+115524us) service_pool.cc:227] Handling call
0925 03:04:52.363318 (+2111741us) spinlock_profiling.cc:243] Waited 1.22 s on lock 0x7b5400280ad4. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d42ae 00007f437b411c27 00007f4374af9c04 00007f4374af9b92 00007f4374af9ac1 00007f4374af9a3c 00007f4374af8cf2 00007f43748fa94c 00007f43748f9d86 00007f43748f972f
0925 03:04:52.363554 (+ 236us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2806336768}
I20260925 03:04:52.366644 24539 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 935) took 1845 ms. Trace:
I20260925 03:04:52.370460 24539 rpcz_store.cc:276] 0925 03:04:50.520882 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:52.337996 (+1817114us) service_pool.cc:227] Handling call
0925 03:04:52.360169 (+ 22173us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:52.366386 (+ 6217us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:52.366591 (+ 205us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260925 03:04:52.371835 24585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.4:43979, user_credentials={real_user=slave}} blocked reactor thread for 795367us
W20260925 03:04:52.364205 25199 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48424 (request call id 214) took 2961 ms (client timeout 2117 ms). Trace:
I20260925 03:04:52.364742 26508 raft_consensus.cc:3037] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:52.371722 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6b6ec010caf4689b8c90449764f5cf9" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:52.374918 24548 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 933) took 1854 ms. Trace:
W20260925 03:04:52.372432 25199 rpcz_store.cc:269] 0925 03:04:49.403094 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:49.403210 (+ 116us) service_pool.cc:227] Handling call
0925 03:04:52.363853 (+2960643us) spinlock_profiling.cc:243] Waited 1.01 s on lock 0x7b540067e154. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d42ae 00007f437b411c27 00007f4374af9c04 00007f4374af9b92 00007f4374af9ac1 00007f4374af9a3c 00007f4374af8cf2 00007f43748fa94c 00007f43748f9d86 00007f43748f972f
0925 03:04:52.364021 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2319342464}
I20260925 03:04:52.369141 26741 raft_consensus.cc:686] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [term 5 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:52.353968 26450 raft_consensus.cc:504] T 7e49248643df4131a1c42e74244e05c2 P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:52.377250 24548 rpcz_store.cc:276] 0925 03:04:50.520327 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:52.321974 (+1801647us) service_pool.cc:227] Handling call
0925 03:04:52.352899 (+ 30925us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:52.369614 (+ 16715us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:52.369640 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:52.374053 (+ 4413us) raft_consensus.cc:1631] Marking committed up to 3
0925 03:04:52.374100 (+ 47us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:52.374160 (+ 60us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:52.374261 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:52.380851 24546 raft_consensus.cc:2454] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 4.
I20260925 03:04:52.381516 26741 consensus_queue.cc:237] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:52.383714 24546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 940) took 1850 ms. Trace:
I20260925 03:04:52.384253 24546 rpcz_store.cc:276] 0925 03:04:50.532837 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:52.381845 (+1849008us) service_pool.cc:227] Handling call
0925 03:04:52.382580 (+ 735us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:52.383051 (+ 471us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:52.383074 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:52.383146 (+ 72us) raft_consensus.cc:1631] Marking committed up to 2
0925 03:04:52.383210 (+ 64us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:52.383317 (+ 107us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:52.383461 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:52.380992 26551 leader_election.cc:288] T 21c43af5e6844fdaacc8c625fa01fd8d P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:52.383426 26380 raft_consensus.cc:3037] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:04:52.388247 24585 leader_election.cc:302] T 04e068a0f80941dc9138cee063f53bab P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, eeab93abfdb141a895002ec9f312296c; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:52.361107 26764 raft_consensus.cc:504] T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:52.392426 24540 raft_consensus.cc:1226] T edcf6e6a422d4d198d4fbb0f7d3d845d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 3. Current term is 4. Ops: [3.2-3.2]
W20260925 03:04:52.369426 25162 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 379424us
I20260925 03:04:52.357357 26766 raft_consensus.cc:504] T ceab6b0fccd54e909a64f66fe81a9a1f P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:52.394460 24540 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 937) took 1872 ms. Trace:
I20260925 03:04:52.404161 24540 rpcz_store.cc:276] 0925 03:04:50.522479 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:52.345197 (+1822718us) service_pool.cc:227] Handling call
0925 03:04:52.369143 (+ 23946us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:04:52.393779 (+ 24636us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:52.394391 (+ 612us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:52.359088 26679 raft_consensus.cc:504] T 71f461a6e2ff434993b78a2d077f0abd P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:52.405468 24548 raft_consensus.cc:1261] T 43366a0a2f9d4344bf4e37ffd290ef8a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260925 03:04:52.410118 24548 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 939) took 1887 ms. Trace:
I20260925 03:04:52.411818 24548 rpcz_store.cc:276] 0925 03:04:50.523028 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:52.385799 (+1862771us) service_pool.cc:227] Handling call
0925 03:04:52.389440 (+ 3641us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:52.409844 (+ 20404us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:52.410064 (+ 220us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:52.421603 24543 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 938) took 1898 ms. Trace:
I20260925 03:04:52.424252 24543 rpcz_store.cc:276] 0925 03:04:50.522766 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:52.386758 (+1863992us) service_pool.cc:227] Handling call
0925 03:04:52.407281 (+ 20523us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:52.421213 (+ 13932us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:52.421247 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:52.421305 (+ 58us) raft_consensus.cc:1631] Marking committed up to 5
0925 03:04:52.421342 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:52.421407 (+ 65us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:52.421544 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:52.405311 26378 consensus_queue.cc:260] T edcf6e6a422d4d198d4fbb0f7d3d845d P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:52.428493 26378 raft_consensus.cc:3037] T edcf6e6a422d4d198d4fbb0f7d3d845d P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:52.429114 26743 consensus_queue.cc:1046] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.037s
I20260925 03:04:52.430111 24541 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 934) took 1909 ms. Trace:
I20260925 03:04:52.430492 24541 rpcz_store.cc:276] 0925 03:04:50.520599 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:52.334323 (+1813724us) service_pool.cc:227] Handling call
0925 03:04:52.375451 (+ 41128us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:52.416518 (+ 41067us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:52.416546 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:52.429833 (+ 13287us) raft_consensus.cc:1631] Marking committed up to 2
0925 03:04:52.429879 (+ 46us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:52.429937 (+ 58us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:52.430040 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:52.414719 26241 raft_consensus.cc:482] T a7aa47c48404465b94a36fd819fe3c7a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:52.433804 24586 leader_election.cc:302] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:52.433527 26241 raft_consensus.cc:504] T a7aa47c48404465b94a36fd819fe3c7a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:52.434805 26386 raft_consensus.cc:2781] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Leader election won for term 6
I20260925 03:04:52.435374 26386 raft_consensus.cc:686] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [term 6 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:52.436532 26386 consensus_queue.cc:237] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:52.438867 24472 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1074) took 2933 ms. Trace:
I20260925 03:04:52.439260 24472 rpcz_store.cc:276] 0925 03:04:49.505646 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:49.803255 (+297609us) service_pool.cc:227] Handling call
0925 03:04:49.803595 (+ 340us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:51.446074 (+1642479us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:52.438529 (+992455us) spinlock_profiling.cc:243] Waited 1.64 s on lock 0x7b54001923d4. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d0f51 00007f43764cf99e 00007f437b4112e9 00007f4374af7684 00007f4374af7612 00007f4374af7541 00007f4374af74bc 00007f4374af6772 00007f43748fa94c 00007f43748f9d86
0925 03:04:52.438803 (+ 274us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3772880128}
I20260925 03:04:52.440079 24586 leader_election.cc:302] T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:52.441038 26743 raft_consensus.cc:2781] T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260925 03:04:52.442801 26743 raft_consensus.cc:482] T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting leader election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:52.443138 26743 raft_consensus.cc:3037] T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Advancing to term 5
W20260925 03:04:52.433684 24585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.4:43979, user_credentials={real_user=slave}} blocked reactor thread for 45524.9us
I20260925 03:04:52.414161 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15861cd28e354598a3c6bfc4337c34bb" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:52.377942 26756 consensus_queue.cc:260] T cc2df40db08649c9a75e90000b878559 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:52.459340 26245 raft_consensus.cc:3032] T 550118bc1a0641ce9803bdbf43f2069f P eeab93abfdb141a895002ec9f312296c [term 4 LEADER]: Stepping down as leader of term 4
I20260925 03:04:52.459929 26245 raft_consensus.cc:729] T 550118bc1a0641ce9803bdbf43f2069f P eeab93abfdb141a895002ec9f312296c [term 4 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:52.461277 26245 consensus_queue.cc:260] T 550118bc1a0641ce9803bdbf43f2069f P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:52.462641 26245 raft_consensus.cc:3037] T 550118bc1a0641ce9803bdbf43f2069f P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:52.394435 26720 consensus_queue.cc:260] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:52.465638 26720 raft_consensus.cc:3037] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:04:52.457437 24546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 941) took 1919 ms. Trace:
I20260925 03:04:52.470589 24586 leader_election.cc:302] T 5278a60f97da4e10aecd50a8d9a9a43c P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:52.387132 26737 raft_consensus.cc:504] T 29637f5a4a504ce6ba29033ccd0cd716 P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:52.469206 24546 rpcz_store.cc:276] 0925 03:04:50.537722 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:52.388426 (+1850704us) service_pool.cc:227] Handling call
0925 03:04:52.444359 (+ 55933us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:52.452854 (+ 8495us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:52.452880 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:52.457140 (+ 4260us) raft_consensus.cc:1631] Marking committed up to 3
0925 03:04:52.457189 (+ 49us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:52.457254 (+ 65us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:52.457372 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:52.467209 26551 raft_consensus.cc:2781] T 04e068a0f80941dc9138cee063f53bab P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader election won for term 3
I20260925 03:04:52.476648 26552 raft_consensus.cc:3032] T 1414dcfe53f64a78ae413531eb8ec117 P eeab93abfdb141a895002ec9f312296c [term 2 LEADER]: Stepping down as leader of term 2
I20260925 03:04:52.476868 26241 leader_election.cc:288] T a7aa47c48404465b94a36fd819fe3c7a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:52.477424 25163 leader_election.cc:302] T e92f377a79354a6ebb4eec67303c13e3 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:04:52.477957 26738 raft_consensus.cc:2781] T 5278a60f97da4e10aecd50a8d9a9a43c P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260925 03:04:52.476089 24539 raft_consensus.cc:2454] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 4.
I20260925 03:04:52.482847 26738 raft_consensus.cc:482] T 5278a60f97da4e10aecd50a8d9a9a43c P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting leader election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:52.487151 26738 raft_consensus.cc:3037] T 5278a60f97da4e10aecd50a8d9a9a43c P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:52.487623 26749 consensus_queue.cc:1035] T 25102ba08e4e49388f6176badcd10a25 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260925 03:04:52.488338 26551 raft_consensus.cc:686] T 04e068a0f80941dc9138cee063f53bab P eeab93abfdb141a895002ec9f312296c [term 3 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:52.480500 26552 raft_consensus.cc:729] T 1414dcfe53f64a78ae413531eb8ec117 P eeab93abfdb141a895002ec9f312296c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:52.489663 26551 consensus_queue.cc:237] T 04e068a0f80941dc9138cee063f53bab P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
W20260925 03:04:52.498224 24467 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1293) took 2161 ms (client timeout 1637 ms). Trace:
W20260925 03:04:52.498723 24467 rpcz_store.cc:269] 0925 03:04:50.336321 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:52.497920 (+2161599us) service_pool.cc:214] Skipping call since client already timed out
0925 03:04:52.498108 (+ 188us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20260925 03:04:52.499817 24467 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1286) took 2235 ms (client timeout 1932 ms). Trace:
W20260925 03:04:52.500200 24467 rpcz_store.cc:269] 0925 03:04:50.264742 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:52.499609 (+2234867us) service_pool.cc:214] Skipping call since client already timed out
0925 03:04:52.499740 (+ 131us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20260925 03:04:52.501211 24467 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1285) took 2250 ms (client timeout 1966 ms). Trace:
W20260925 03:04:52.501602 24467 rpcz_store.cc:269] 0925 03:04:50.250914 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:52.501034 (+2250120us) service_pool.cc:214] Skipping call since client already timed out
0925 03:04:52.501149 (+ 115us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20260925 03:04:52.502418 24467 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1273) took 2256 ms (client timeout 2019 ms). Trace:
W20260925 03:04:52.504068 24467 rpcz_store.cc:269] 0925 03:04:50.245689 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:52.502235 (+2256546us) service_pool.cc:214] Skipping call since client already timed out
0925 03:04:52.502360 (+ 125us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20260925 03:04:52.510155 24467 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1299) took 2165 ms (client timeout 1923 ms). Trace:
W20260925 03:04:52.512348 24467 rpcz_store.cc:269] 0925 03:04:50.346194 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:52.509898 (+2163704us) service_pool.cc:214] Skipping call since client already timed out
0925 03:04:52.510073 (+ 175us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20260925 03:04:52.514451 24467 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1291) took 2184 ms (client timeout 1939 ms). Trace:
I20260925 03:04:52.506127 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21c43af5e6844fdaacc8c625fa01fd8d" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:52.516521 24620 raft_consensus.cc:2379] T 21c43af5e6844fdaacc8c625fa01fd8d P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 4: Already voted for candidate eeab93abfdb141a895002ec9f312296c in this term.
W20260925 03:04:52.517554 24467 rpcz_store.cc:269] 0925 03:04:50.330445 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:52.514284 (+2183839us) service_pool.cc:214] Skipping call since client already timed out
0925 03:04:52.514402 (+ 118us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20260925 03:04:52.519390 24467 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1279) took 2269 ms (client timeout 2027 ms). Trace:
I20260925 03:04:52.505317 26437 raft_consensus.cc:482] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:52.515031 24465 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1050) took 3332 ms. Trace:
W20260925 03:04:52.522897 24467 rpcz_store.cc:269] 0925 03:04:50.249476 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:52.519182 (+2269706us) service_pool.cc:214] Skipping call since client already timed out
0925 03:04:52.519336 (+ 154us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260925 03:04:52.524204 24465 rpcz_store.cc:276] 0925 03:04:49.182148 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:49.185427 (+ 3279us) service_pool.cc:227] Handling call
0925 03:04:49.185722 (+ 295us) raft_consensus.cc:1457] Updating replica for 2 ops
0925 03:04:50.433564 (+1247842us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:52.514557 (+2080993us) spinlock_profiling.cc:243] Waited 1.21 s on lock 0x7b54003eb7d4. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d0f51 00007f43764cf99e 00007f437b4112e9 00007f4374af7684 00007f4374af7612 00007f4374af7541 00007f4374af74bc 00007f4374af6772 00007f43748fa94c 00007f43748f9d86
0925 03:04:52.514870 (+ 313us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2787399424}
I20260925 03:04:52.526666 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e068a0f80941dc9138cee063f53bab" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
W20260925 03:04:52.501350 26775 log.cc:924] Time spent T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.125s user 0.001s sys 0.000s
W20260925 03:04:52.536638 24467 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1276) took 2290 ms (client timeout 2035 ms). Trace:
I20260925 03:04:52.521699 26646 raft_consensus.cc:3037] T 8ac466793d094325ace224708b2158ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:52.540055 26777 raft_consensus.cc:482] T b4ebd2b043024a548005c8b6b00b0449 P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:52.540671 26637 raft_consensus.cc:482] T 3d89396576884f7c8901dba54f783947 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:52.541368 26637 raft_consensus.cc:504] T 3d89396576884f7c8901dba54f783947 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:52.543562 26637 leader_election.cc:288] T 3d89396576884f7c8901dba54f783947 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:52.544445 26689 raft_consensus.cc:482] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:52.545208 26689 raft_consensus.cc:504] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:52.547534 26689 leader_election.cc:288] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:52.551061 26293 raft_consensus.cc:3037] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Advancing to term 8
I20260925 03:04:52.553248 25193 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49794 (request call id 216) took 2791 ms. Trace:
I20260925 03:04:52.553643 25193 rpcz_store.cc:276] 0925 03:04:49.761353 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:49.767475 (+ 6122us) service_pool.cc:227] Handling call
0925 03:04:51.560859 (+1793384us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:51.566239 (+ 5380us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:52.553005 (+986766us) spinlock_profiling.cc:243] Waited 1.79 s on lock 0x7b54005bf750. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764cfa09 00007f437b4112e9 00007f4374af7684 00007f4374af7612 00007f4374af7541 00007f4374af74bc 00007f4374af6772 00007f43748fa94c 00007f43748f9d86 00007f43748f972f
0925 03:04:52.553159 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4114941824}
I20260925 03:04:52.558616 26743 raft_consensus.cc:504] T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:52.559690 24586 leader_election.cc:302] T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:52.560547 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1e85248d3b04b979dd080caeeb02fdb" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:52.561221 24621 raft_consensus.cc:2454] T b1e85248d3b04b979dd080caeeb02fdb P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 5.
I20260925 03:04:52.561342 26508 raft_consensus.cc:2781] T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Leader election won for term 6
I20260925 03:04:52.562055 26508 raft_consensus.cc:686] T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c [term 6 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:52.562567 26744 consensus_queue.cc:1046] T 367277daf8d04778b3fa4ed94d71580d P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true }, Status: INVALID_TERM, Last received: 4.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260925 03:04:52.563565 26764 leader_election.cc:288] T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:52.563230 26508 consensus_queue.cc:237] T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:52.564304 26764 raft_consensus.cc:3032] T 367277daf8d04778b3fa4ed94d71580d P eeab93abfdb141a895002ec9f312296c [term 4 LEADER]: Stepping down as leader of term 4
I20260925 03:04:52.564710 26764 raft_consensus.cc:729] T 367277daf8d04778b3fa4ed94d71580d P eeab93abfdb141a895002ec9f312296c [term 4 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
W20260925 03:04:52.539148 24467 rpcz_store.cc:269] 0925 03:04:50.246506 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:52.524647 (+2278141us) service_pool.cc:214] Skipping call since client already timed out
0925 03:04:52.536566 (+ 11919us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260925 03:04:52.570742 24543 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 948) took 1003 ms. Trace:
I20260925 03:04:52.571136 24543 rpcz_store.cc:276] 0925 03:04:51.567201 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:52.569859 (+1002658us) service_pool.cc:227] Handling call
0925 03:04:52.570225 (+ 366us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:52.570449 (+ 224us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:52.570472 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:52.570516 (+ 44us) raft_consensus.cc:1631] Marking committed up to 2
0925 03:04:52.570543 (+ 27us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:52.570588 (+ 45us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:52.570685 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:52.555629 24437 service_pool.cc:140] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:55562 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1278)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1287)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1292)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1296)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1283)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1289)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 1080)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 1087)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 1084)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 1097)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 1098)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1308)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1311)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1305)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1312)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1313)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1314)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1315)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1304)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1317)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1306)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1302)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1309)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1318)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38920 (request call id 131)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1310)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1307)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1316)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1303)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38920 (request call id 128)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38920 (request call id 129)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38920 (request call id 130)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55560 (request call id 1281)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55560 (request call id 1297)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55560 (request call id 1301)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1078)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1079)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1081)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1082)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1083)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1085)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1086)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1088)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1089)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1090)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1091)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1092)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1093)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1094)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1095)
I20260925 03:04:52.562924 26450 leader_election.cc:288] T 7e49248643df4131a1c42e74244e05c2 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:52.572578 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e92f377a79354a6ebb4eec67303c13e3" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:52.552079 26738 raft_consensus.cc:504] T 5278a60f97da4e10aecd50a8d9a9a43c P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:52.526360 26437 raft_consensus.cc:3037] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Advancing to term 7
W20260925 03:04:52.594321 24465 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1284) took 2324 ms (client timeout 2085 ms). Trace:
I20260925 03:04:52.594499 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43366a0a2f9d4344bf4e37ffd290ef8a" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
W20260925 03:04:52.594913 24465 rpcz_store.cc:269] 0925 03:04:50.250688 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:52.536579 (+2285891us) service_pool.cc:214] Skipping call since client already timed out
0925 03:04:52.575135 (+ 38556us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20260925 03:04:52.594902 25163 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 121929us
I20260925 03:04:52.565616 26743 leader_election.cc:288] T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:52.326754 24439 service_pool.cc:140] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:55562 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1295)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1280)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1298)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1294)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1286)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1285)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1273)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1299)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1291)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1279)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1276)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1284)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1278)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1287)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1292)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1296)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1283)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1289)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 1080)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 1087)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 1084)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 1097)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 1098)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38920 (request call id 128)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38920 (request call id 129)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38920 (request call id 130)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55560 (request call id 1281)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55560 (request call id 1297)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55560 (request call id 1301)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1078)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1079)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1081)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1082)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1083)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1085)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1086)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1088)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1089)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1090)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1091)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1092)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1093)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1094)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1095)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1096)
I20260925 03:04:52.616984 26530 raft_consensus.cc:482] T 8ac466793d094325ace224708b2158ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:52.617591 26530 raft_consensus.cc:504] T 8ac466793d094325ace224708b2158ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:52.617228 24439 service_pool.cc:140] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:55562 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1292)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1296)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1283)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1289)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 1080)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 1087)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 1084)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 1097)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 1098)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1308)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1311)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1305)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1312)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1313)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1314)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1315)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1304)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1317)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1321)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1306)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1302)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1309)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1318)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38920 (request call id 131)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1310)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1319)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1307)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1320)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1316)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1303)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38920 (request call id 128)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38920 (request call id 129)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38920 (request call id 130)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55560 (request call id 1281)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55560 (request call id 1297)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55560 (request call id 1301)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1078)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1079)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1081)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1082)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1083)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1085)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1086)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1088)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1089)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1090)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1091)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1092)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1093)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1094)
I20260925 03:04:52.598570 24621 raft_consensus.cc:1226] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 4. Current term is 5. Ops: []
I20260925 03:04:52.540741 26777 raft_consensus.cc:504] T b4ebd2b043024a548005c8b6b00b0449 P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:52.623102 26777 leader_election.cc:288] T b4ebd2b043024a548005c8b6b00b0449 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:52.601879 24544 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 1428) took 2640 ms. Trace:
I20260925 03:04:52.624468 26712 raft_consensus.cc:3032] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c [term 5 LEADER]: Stepping down as leader of term 5
I20260925 03:04:52.624938 26712 raft_consensus.cc:729] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c [term 5 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:52.625486 26679 leader_election.cc:288] T 71f461a6e2ff434993b78a2d077f0abd P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:52.626636 24615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35992316cb8d4c028333388eff02ecde" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
W20260925 03:04:52.627363 26726 log.cc:924] Time spent T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 2.189s user 0.013s sys 0.010s
I20260925 03:04:52.559376 26766 leader_election.cc:288] T ceab6b0fccd54e909a64f66fe81a9a1f P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:52.604462 25199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6b6ec010caf4689b8c90449764f5cf9" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
W20260925 03:04:52.629207 26778 log.cc:924] Time spent T a7aa47c48404465b94a36fd819fe3c7a P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.206s user 0.001s sys 0.000s
I20260925 03:04:52.629431 25199 raft_consensus.cc:2454] T e6b6ec010caf4689b8c90449764f5cf9 P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 4.
I20260925 03:04:52.604190 25193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4abe0ea757e4886be4945c5001e529a" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
W20260925 03:04:52.608844 26774 log.cc:924] Time spent T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.189s user 0.000s sys 0.003s
W20260925 03:04:52.610260 26768 log.cc:924] Time spent T 48ca6ccd98194cf2af7b5278d15bf4ea P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.189s user 0.000s sys 0.000s
I20260925 03:04:52.613210 24547 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 929) took 2673 ms. Trace:
I20260925 03:04:52.639356 24547 rpcz_store.cc:276] 0925 03:04:49.939691 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:51.484359 (+1544668us) service_pool.cc:227] Handling call
0925 03:04:51.485023 (+ 664us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:04:51.514692 (+ 29669us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:51.514754 (+ 62us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:04:51.515552 (+ 798us) log.cc:842] Serialized 26 byte log entry
0925 03:04:51.544797 (+ 29245us) raft_consensus.cc:1631] Marking committed up to 2
0925 03:04:51.544828 (+ 31us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:51.544892 (+ 64us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:04:52.612950 (+1068058us) raft_consensus.cc:1671] finished
0925 03:04:52.612971 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:52.613150 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":28922,"raft.queue_time_us":62142,"raft.run_cpu_time_us":208,"raft.run_wall_time_us":209,"thread_start_us":28048,"threads_started":1,"wal-append.queue_time_us":890141}
I20260925 03:04:52.613895 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "420163a6cdce412ebc21c25fe3835638" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:52.612529 24437 service_pool.cc:140] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:55562 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1292)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1296)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1283)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1289)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 1080)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 1087)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 1084)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 1097)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 1098)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1308)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1311)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1305)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1312)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1313)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1314)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1315)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1304)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1317)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1321)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1306)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1302)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1309)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1318)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38920 (request call id 131)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1310)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1319)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1307)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1320)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1316)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1303)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38920 (request call id 128)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38920 (request call id 129)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38920 (request call id 130)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55560 (request call id 1281)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55560 (request call id 1297)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55560 (request call id 1301)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1078)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1079)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1081)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1082)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1083)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1085)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1086)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1088)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1089)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1090)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1091)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1092)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1093)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1094)
I20260925 03:04:52.610105 26738 leader_election.cc:288] T 5278a60f97da4e10aecd50a8d9a9a43c P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:52.623688 24544 rpcz_store.cc:276] 0925 03:04:49.961161 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:51.595419 (+1634258us) service_pool.cc:227] Handling call
0925 03:04:51.698678 (+103259us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:04:51.714985 (+ 16307us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:51.715014 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:04:51.735148 (+ 20134us) log.cc:842] Serialized 26 byte log entry
0925 03:04:52.250297 (+515149us) raft_consensus.cc:1631] Marking committed up to 4
0925 03:04:52.250340 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:52.250418 (+ 78us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:04:52.601635 (+351217us) raft_consensus.cc:1671] finished
0925 03:04:52.601652 (+ 17us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:52.601819 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":894,"raft.run_cpu_time_us":305,"raft.run_wall_time_us":82511,"thread_start_us":143629,"threads_started":1,"wal-append.queue_time_us":527043}
I20260925 03:04:52.605610 25198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c81a1b3b616492ab6c964f45af36077" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
W20260925 03:04:52.630420 24465 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1287) took 2351 ms (client timeout 2117 ms). Trace:
W20260925 03:04:52.642422 24465 rpcz_store.cc:269] 0925 03:04:50.278872 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:52.595839 (+2316967us) service_pool.cc:214] Skipping call since client already timed out
0925 03:04:52.630347 (+ 34508us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260925 03:04:52.644081 26738 raft_consensus.cc:482] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:52.637125 26658 consensus_queue.cc:1035] T 43366a0a2f9d4344bf4e37ffd290ef8a P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260925 03:04:52.637686 24622 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38570 (request call id 1475) took 2676 ms. Trace:
W20260925 03:04:52.633241 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 71049.5us
I20260925 03:04:52.633806 24439 service_pool.cc:140] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:55562 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1292)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1296)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1283)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1289)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 1080)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 1087)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 1084)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 1097)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 1098)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1308)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1311)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1305)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1312)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1313)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1314)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1315)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1304)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1317)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1321)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1306)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1302)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1309)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1318)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38920 (request call id 131)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1310)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1319)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1307)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1320)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 1101)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1316)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1303)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38920 (request call id 128)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38920 (request call id 129)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38920 (request call id 130)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55560 (request call id 1281)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55560 (request call id 1297)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55560 (request call id 1301)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1078)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1079)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1081)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1082)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1083)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1085)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1086)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1088)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1089)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1090)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1091)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1092)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1093)
I20260925 03:04:52.558691 26737 leader_election.cc:288] T 29637f5a4a504ce6ba29033ccd0cd716 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
W20260925 03:04:52.654017 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 168860us
I20260925 03:04:52.644654 26738 raft_consensus.cc:504] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:52.632550 26450 raft_consensus.cc:2781] T e92f377a79354a6ebb4eec67303c13e3 P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260925 03:04:52.559463 26719 leader_election.cc:288] T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:52.658017 24615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abd5b8dc1b7e49e3aa9af900d73ff0e7" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:52.659921 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4abe0ea757e4886be4945c5001e529a" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:52.658551 26766 raft_consensus.cc:482] T 550118bc1a0641ce9803bdbf43f2069f P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:52.666898 24439 service_pool.cc:140] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:55562 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1296)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1283)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1289)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 1080)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 1087)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 1084)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 1097)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 1098)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1308)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1311)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1305)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1312)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1313)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1314)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1315)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1304)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1317)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1321)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1306)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1302)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1309)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1318)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38920 (request call id 131)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1310)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1319)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1322)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 1102)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1307)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1320)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 1101)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1316)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1303)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38920 (request call id 128)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38920 (request call id 129)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38920 (request call id 130)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55560 (request call id 1281)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55560 (request call id 1297)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55560 (request call id 1301)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1078)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1079)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1081)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1082)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1083)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1085)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1086)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1088)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1089)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1090)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1091)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1092)
I20260925 03:04:52.650271 24622 rpcz_store.cc:276] 0925 03:04:49.961596 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:49.970780 (+ 9184us) service_pool.cc:227] Handling call
0925 03:04:49.971076 (+ 296us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:04:49.991918 (+ 20842us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:49.991948 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:04:50.006921 (+ 14973us) log.cc:842] Serialized 26 byte log entry
0925 03:04:50.240846 (+233925us) raft_consensus.cc:1631] Marking committed up to 4
0925 03:04:50.240889 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:50.240967 (+ 78us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:04:52.626417 (+2385450us) spinlock_profiling.cc:243] Waited 14.7 ms on lock 0x7b44000232cc. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f43733ecf11 00007f43734baa95 00007f43734bb4b2 00007f4376163511 00007f437615ee1a 00007f4376152bee 00007f4376151f36 00007f4376151b58 00007f437615b551 00007f437615b509 00007f437615b499
0925 03:04:52.630187 (+ 3770us) raft_consensus.cc:1671] finished
0925 03:04:52.630214 (+ 27us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:52.637483 (+ 7269us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":14525,"raft.queue_time_us":33721,"raft.run_cpu_time_us":204,"raft.run_wall_time_us":29093,"spinlock_wait_cycles":33833344,"thread_start_us":118621,"threads_started":1,"wal-append.queue_time_us":339229}
I20260925 03:04:52.676889 26758 raft_consensus.cc:482] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260925 03:04:52.681456 26769 log.cc:924] Time spent T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5: Append to log took a long time: real 0.230s user 0.001s sys 0.000s
W20260925 03:04:52.684484 24466 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1290) took 2385 ms (client timeout 1556 ms). Trace:
W20260925 03:04:52.684938 24466 rpcz_store.cc:269] 0925 03:04:50.299427 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:50.438677 (+139250us) service_pool.cc:227] Handling call
0925 03:04:52.684178 (+2245501us) spinlock_profiling.cc:243] Waited 32 ms on lock 0x7b540026fcd4. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d42ae 00007f437b411c27 00007f4374af9c04 00007f4374af9b92 00007f4374af9ac1 00007f4374af9a3c 00007f4374af8cf2 00007f43748fa94c 00007f43748f9d86 00007f43748f972f
0925 03:04:52.684314 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":73699200}
I20260925 03:04:52.625947 26712 consensus_queue.cc:260] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:52.685489 26437 raft_consensus.cc:504] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:52.685976 24615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:52.686359 26712 raft_consensus.cc:3037] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:04:52.688279 26437 leader_election.cc:288] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:52.690670 24618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e5819c15eac4544a57e5ee6c9d4f117" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:52.692052 24618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c88cbd39e01f44628a094af516be2833" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:52.693056 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be4e0aa984f4f2b81a58e382e434ad5" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:52.693173 24618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e92f377a79354a6ebb4eec67303c13e3" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:52.693409 24614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c23337a7cb74184a0e57b7095cb6a11" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:52.693813 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e725b82985746b39c0cc8f671c95353" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:52.694243 24618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7aa47c48404465b94a36fd819fe3c7a" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:52.694547 24614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48ca6ccd98194cf2af7b5278d15bf4ea" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:52.694866 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "550118bc1a0641ce9803bdbf43f2069f" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:52.695493 24620 raft_consensus.cc:2454] T 550118bc1a0641ce9803bdbf43f2069f P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 5.
I20260925 03:04:52.695775 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e068a0f80941dc9138cee063f53bab" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:52.658771 26530 leader_election.cc:288] T 8ac466793d094325ace224708b2158ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:52.696308 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06f4a7eaf6534e4986c9fd35911038e4" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:52.697053 24616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f1fac028a144058b92331c0d9b44ebf" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
W20260925 03:04:52.697312 24468 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1274) took 2451 ms (client timeout 1714 ms). Trace:
I20260925 03:04:52.685000 25198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e068a0f80941dc9138cee063f53bab" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43"
W20260925 03:04:52.698800 26771 log.cc:924] Time spent T 0b5df303cd6a4d59aca5c18bbbff24ed P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.275s user 0.001s sys 0.001s
I20260925 03:04:52.700582 26758 raft_consensus.cc:504] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:52.702536 26738 leader_election.cc:288] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:52.702710 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:52.703356 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceab6b0fccd54e909a64f66fe81a9a1f" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:52.695698 24614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c85f5d1d3f004cf5817bb594fd8e6469" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
W20260925 03:04:52.704165 24468 rpcz_store.cc:269] 0925 03:04:50.245981 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:52.697095 (+2451114us) service_pool.cc:214] Skipping call since client already timed out
0925 03:04:52.697247 (+ 152us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260925 03:04:52.565689 26764 consensus_queue.cc:260] T 367277daf8d04778b3fa4ed94d71580d P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:52.704396 24620 raft_consensus.cc:2454] T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 6.
I20260925 03:04:52.658470 26450 raft_consensus.cc:482] T e92f377a79354a6ebb4eec67303c13e3 P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:52.690384 24437 service_pool.cc:140] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:55562 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1296)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1283)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1289)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 1080)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 1087)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 1084)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 1097)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 1098)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1308)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1311)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1305)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1312)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1313)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1314)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1315)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1304)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1317)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1321)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1306)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1302)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1309)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1318)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38920 (request call id 131)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1310)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1319)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1322)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 1102)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1323)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1307)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1320)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 1101)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1316)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1303)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38920 (request call id 128)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38920 (request call id 129)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38920 (request call id 130)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55560 (request call id 1281)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55560 (request call id 1297)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55560 (request call id 1301)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1078)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1079)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1081)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1082)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1083)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1085)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1086)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1088)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1089)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1090)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1091)
I20260925 03:04:52.711323 25198 raft_consensus.cc:3037] T 04e068a0f80941dc9138cee063f53bab P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 FOLLOWER]: Advancing to term 4
W20260925 03:04:52.716972 24468 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1283) took 2463 ms (client timeout 2446 ms). Trace:
W20260925 03:04:52.717439 24468 rpcz_store.cc:269] 0925 03:04:50.250458 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:52.711015 (+2460557us) service_pool.cc:214] Skipping call since client already timed out
0925 03:04:52.714146 (+ 3131us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260925 03:04:52.718542 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ac285175464fbda67211d9574db415" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:52.718951 24470 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1066) took 3478 ms. Trace:
I20260925 03:04:52.719523 24468 raft_consensus.cc:2396] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
W20260925 03:04:52.720007 24468 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1289) took 2425 ms (client timeout 2942 ms). Trace:
I20260925 03:04:52.714066 26386 consensus_queue.cc:1046] T 367277daf8d04778b3fa4ed94d71580d P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.007s
I20260925 03:04:52.721442 25199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ac466793d094325ace224708b2158ed" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
W20260925 03:04:52.721664 24467 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1278) took 2472 ms (client timeout 2132 ms). Trace:
I20260925 03:04:52.722088 25199 raft_consensus.cc:2454] T 8ac466793d094325ace224708b2158ed P 1bc1f2e62ab44af995165bd5116a9d43 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 5.
I20260925 03:04:52.723026 25193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e92f377a79354a6ebb4eec67303c13e3" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
W20260925 03:04:52.723215 24471 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1295) took 2386 ms (client timeout 1656 ms). Trace:
W20260925 03:04:52.723615 24471 rpcz_store.cc:269] 0925 03:04:50.336957 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:52.713337 (+2376380us) service_pool.cc:214] Skipping call since client already timed out
0925 03:04:52.723149 (+ 9812us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260925 03:04:52.724452 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:52.728089 26758 leader_election.cc:288] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:52.728608 26766 raft_consensus.cc:504] T 550118bc1a0641ce9803bdbf43f2069f P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:52.730008 24615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ac466793d094325ace224708b2158ed" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
W20260925 03:04:52.722098 24467 rpcz_store.cc:269] 0925 03:04:50.249199 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:52.721473 (+2472274us) service_pool.cc:214] Skipping call since client already timed out
0925 03:04:52.721604 (+ 131us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260925 03:04:52.731923 24467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b627ecb4a5384ef392aa695e06599c39" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:52.732261 24542 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 930) took 2212 ms. Trace:
I20260925 03:04:52.732347 26530 raft_consensus.cc:482] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260925 03:04:52.729264 24469 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1275) took 2467 ms (client timeout 1693 ms). Trace:
I20260925 03:04:52.719318 24470 rpcz_store.cc:276] 0925 03:04:49.240887 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:49.241025 (+ 138us) service_pool.cc:227] Handling call
0925 03:04:49.366353 (+125328us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:50.477000 (+1110647us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:52.718499 (+2241499us) spinlock_profiling.cc:243] Waited 1.08 s on lock 0x7b540026f7d4. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d0f51 00007f43764cf99e 00007f437b4112e9 00007f4374af7684 00007f4374af7612 00007f4374af7541 00007f4374af74bc 00007f4374af6772 00007f43748fa94c 00007f43748f9d86
0925 03:04:52.718760 (+ 261us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2488142208}
I20260925 03:04:52.733714 24613 raft_consensus.cc:2379] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1bc1f2e62ab44af995165bd5116a9d43 in current term 7: Already voted for candidate eeab93abfdb141a895002ec9f312296c in this term.
I20260925 03:04:52.734016 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e725b82985746b39c0cc8f671c95353" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
W20260925 03:04:52.733100 24469 rpcz_store.cc:269] 0925 03:04:50.246244 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:52.709414 (+2463170us) service_pool.cc:214] Skipping call since client already timed out
0925 03:04:52.713278 (+ 3864us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260925 03:04:52.732663 24467 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 1087) took 1228 ms. Trace:
I20260925 03:04:52.736321 24467 rpcz_store.cc:276] 0925 03:04:51.503876 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:52.731662 (+1227786us) service_pool.cc:227] Handling call
0925 03:04:52.732609 (+ 947us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:52.734640 24470 raft_consensus.cc:3037] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Advancing to term 6
W20260925 03:04:52.720356 24468 rpcz_store.cc:269] 0925 03:04:50.295073 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:52.718156 (+2423083us) service_pool.cc:227] Handling call
0925 03:04:52.719932 (+ 1776us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:52.741047 25198 raft_consensus.cc:2454] T 04e068a0f80941dc9138cee063f53bab P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 4.
W20260925 03:04:52.742291 24511 connection.cc:232] RPC call timeout handler was delayed by -2.5345086299985269s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15560000000000002s for a response
I20260925 03:04:52.742513 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71f461a6e2ff434993b78a2d077f0abd" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:52.743558 24613 raft_consensus.cc:2396] T 71f461a6e2ff434993b78a2d077f0abd P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1bc1f2e62ab44af995165bd5116a9d43 for term 7 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260925 03:04:52.745913 26682 raft_consensus.cc:504] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:04:52.746556 24472 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1294) took 2409 ms (client timeout 1832 ms). Trace:
I20260925 03:04:52.732769 24542 rpcz_store.cc:276] 0925 03:04:50.519464 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:51.552070 (+1032606us) service_pool.cc:227] Handling call
0925 03:04:51.552605 (+ 535us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:04:51.553308 (+ 703us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:51.553338 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:04:51.553955 (+ 617us) log.cc:842] Serialized 26 byte log entry
0925 03:04:52.698015 (+1144060us) spinlock_profiling.cc:243] Waited 34 ms on lock 0x7b44000232cc. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f43733ecf11 00007f43734baa95 00007f43734bb4b2 00007f4376163511 00007f437615ee1a 00007f4376152bee 00007f4376151f36 00007f4376151b58 00007f437615b551 00007f437615b509 00007f437615b499
0925 03:04:52.712491 (+ 14476us) spinlock_profiling.cc:243] Waited 25.4 ms on lock 0x7b18002fb3c0. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f437c4478b1 00007f437c454661 00007f437615f82c 00007f437358a9dc 00007f437c3c1a39 00007f43735a78bb 00007f43735a5cbf 00007f43735a7e22 00007f43761515bb 00007f4376157063 00007f4376157411
0925 03:04:52.712635 (+ 144us) raft_consensus.cc:1631] Marking committed up to 3
0925 03:04:52.712666 (+ 31us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:52.712755 (+ 89us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:04:52.732023 (+ 19268us) raft_consensus.cc:1671] finished
0925 03:04:52.732059 (+ 36us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:52.732204 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":39176,"raft.queue_time_us":27739,"raft.run_cpu_time_us":230,"raft.run_wall_time_us":60196,"spinlock_wait_cycles":136713856,"thread_start_us":89453,"threads_started":1,"wal-append.queue_time_us":869465}
W20260925 03:04:52.747005 24472 rpcz_store.cc:269] 0925 03:04:50.336666 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:52.746350 (+2409684us) service_pool.cc:214] Skipping call since client already timed out
0925 03:04:52.746478 (+ 128us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260925 03:04:52.730788 26766 leader_election.cc:288] T 550118bc1a0641ce9803bdbf43f2069f P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:52.748404 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367277daf8d04778b3fa4ed94d71580d" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
W20260925 03:04:52.752844 24511 connection.cc:232] RPC call timeout handler was delayed by -2.4859086299984483s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16100000000000003s for a response
I20260925 03:04:52.745951 26552 consensus_queue.cc:260] T 1414dcfe53f64a78ae413531eb8ec117 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:04:52.753471 24511 connection.cc:232] RPC call timeout handler was delayed by -2.02960862999862s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2117s for a response
I20260925 03:04:52.753671 26552 raft_consensus.cc:3037] T 1414dcfe53f64a78ae413531eb8ec117 P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:52.754074 26646 raft_consensus.cc:482] T edcf6e6a422d4d198d4fbb0f7d3d845d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:52.754530 26646 raft_consensus.cc:504] T edcf6e6a422d4d198d4fbb0f7d3d845d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:52.756469 26646 leader_election.cc:288] T edcf6e6a422d4d198d4fbb0f7d3d845d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:52.757139 26656 raft_consensus.cc:482] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:52.757668 26646 raft_consensus.cc:482] T b030d84c670b48d5bff337ccc0c77626 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:52.757813 26656 raft_consensus.cc:504] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:52.759097 24469 raft_consensus.cc:2454] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 4.
I20260925 03:04:52.760306 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abd5b8dc1b7e49e3aa9af900d73ff0e7" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:52.761132 24469 raft_consensus.cc:2379] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 5: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:04:52.761876 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7691c6b475428c82d6181d693d26f3" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:52.762923 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d89396576884f7c8901dba54f783947" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:52.763440 24469 raft_consensus.cc:2454] T 3d89396576884f7c8901dba54f783947 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 3.
I20260925 03:04:52.764212 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc2df40db08649c9a75e90000b878559" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:52.757082 26241 raft_consensus.cc:482] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1bc1f2e62ab44af995165bd5116a9d43)
I20260925 03:04:52.748263 26530 raft_consensus.cc:504] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:52.765424 26241 raft_consensus.cc:504] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:52.766006 26656 leader_election.cc:288] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:52.766201 26646 raft_consensus.cc:504] T b030d84c670b48d5bff337ccc0c77626 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:52.766711 26530 leader_election.cc:288] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
W20260925 03:04:52.767531 24511 connection.cc:232] RPC call timeout handler was delayed by -1.283644610999545s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15480000000000002s for a response
W20260925 03:04:52.767822 24511 connection.cc:232] RPC call timeout handler was delayed by -1.2215446110003541s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1617s for a response
W20260925 03:04:52.768034 24511 connection.cc:232] RPC call timeout handler was delayed by -1.1198446109992801s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17300000000000001s for a response
W20260925 03:04:52.768250 24511 connection.cc:232] RPC call timeout handler was delayed by -1.081144611000127s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1773s for a response
W20260925 03:04:52.768508 24511 connection.cc:232] RPC call timeout handler was delayed by -1.0788359989983292s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1804s for a response
I20260925 03:04:52.768635 25198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5aec38e66a947c681e6c2a72b05e6ca" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
W20260925 03:04:52.768983 24511 connection.cc:232] RPC call timeout handler was delayed by -1.0739446110001154s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1781s for a response
W20260925 03:04:52.769537 24511 connection.cc:232] RPC call timeout handler was delayed by -1.0568446109991783s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18000000000000002s for a response
W20260925 03:04:52.769742 24511 connection.cc:232] RPC call timeout handler was delayed by -1.0262446109991288s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1834s for a response
I20260925 03:04:52.764953 24469 raft_consensus.cc:2379] T cc2df40db08649c9a75e90000b878559 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 5: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:04:52.713595 26450 raft_consensus.cc:3037] T e92f377a79354a6ebb4eec67303c13e3 P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:52.768939 24545 raft_consensus.cc:2454] T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 5.
I20260925 03:04:52.773272 24545 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36786 (request call id 947) took 1206 ms. Trace:
I20260925 03:04:52.773653 24545 rpcz_store.cc:276] 0925 03:04:51.566854 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:52.300486 (+733632us) service_pool.cc:227] Handling call
0925 03:04:52.773200 (+472714us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:52.774396 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5f9b503b0ab43999d582015a367441e" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:52.775926 26241 leader_election.cc:288] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:52.757135 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b627ecb4a5384ef392aa695e06599c39" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
W20260925 03:04:52.777382 24466 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1296) took 2433 ms (client timeout 2239 ms). Trace:
I20260925 03:04:52.778086 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f68804eb5a244d87bb395d50683c6012" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
W20260925 03:04:52.778512 24465 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1292) took 2447 ms (client timeout 2150 ms). Trace:
W20260925 03:04:52.778991 24465 rpcz_store.cc:269] 0925 03:04:50.330746 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:52.778274 (+2447528us) service_pool.cc:214] Skipping call since client already timed out
0925 03:04:52.778447 (+ 173us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260925 03:04:52.780030 24465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35992316cb8d4c028333388eff02ecde" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:52.780411 24586 leader_election.cc:302] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
W20260925 03:04:52.781132 24473 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1280) took 2531 ms (client timeout 1807 ms). Trace:
W20260925 03:04:52.781551 24473 rpcz_store.cc:269] 0925 03:04:50.249732 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:52.780938 (+2531206us) service_pool.cc:214] Skipping call since client already timed out
0925 03:04:52.781067 (+ 129us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260925 03:04:52.782503 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c23337a7cb74184a0e57b7095cb6a11" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:52.783726 24473 raft_consensus.cc:2379] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 6: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:04:52.784593 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c88cbd39e01f44628a094af516be2833" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:52.784886 26380 raft_consensus.cc:2781] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Leader election won for term 5
I20260925 03:04:52.786536 26380 raft_consensus.cc:686] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c [term 5 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:52.786875 24465 raft_consensus.cc:2396] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260925 03:04:52.787844 24465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be4e0aa984f4f2b81a58e382e434ad5" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:52.784925 26241 consensus_queue.cc:1046] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.004s
I20260925 03:04:52.789038 24465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1e85248d3b04b979dd080caeeb02fdb" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:52.787446 26380 consensus_queue.cc:237] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:52.790797 24473 raft_consensus.cc:2379] T c88cbd39e01f44628a094af516be2833 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 6: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
W20260925 03:04:52.777787 24466 rpcz_store.cc:269] 0925 03:04:50.343593 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:52.777178 (+2433585us) service_pool.cc:214] Skipping call since client already timed out
0925 03:04:52.777318 (+ 140us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260925 03:04:52.774547 24471 raft_consensus.cc:2454] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 7.
I20260925 03:04:52.797557 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e8af2641e124f9589c5ccfdac785f63" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:52.797191 24470 raft_consensus.cc:2454] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 6.
I20260925 03:04:52.798944 24470 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 1084) took 1296 ms. Trace:
I20260925 03:04:52.798820 24467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f68804eb5a244d87bb395d50683c6012" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:52.798380 24471 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 1080) took 1297 ms. Trace:
W20260925 03:04:52.797196 24464 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1298) took 2453 ms (client timeout 1804 ms). Trace:
I20260925 03:04:52.799415 24470 rpcz_store.cc:276] 0925 03:04:51.502687 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:52.733761 (+1231074us) service_pool.cc:227] Handling call
0925 03:04:52.798869 (+ 65108us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:52.799610 24471 rpcz_store.cc:276] 0925 03:04:51.501060 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:52.724187 (+1223127us) service_pool.cc:227] Handling call
0925 03:04:52.798315 (+ 74128us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:52.789752 26241 consensus_queue.cc:1046] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20260925 03:04:52.801054 24464 rpcz_store.cc:269] 0925 03:04:50.344151 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:52.771729 (+2427578us) service_pool.cc:214] Skipping call since client already timed out
0925 03:04:52.797132 (+ 25403us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20260925 03:04:52.801546 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 49048.7us
I20260925 03:04:52.786037 26646 leader_election.cc:288] T b030d84c670b48d5bff337ccc0c77626 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:52.890708 24472 raft_consensus.cc:2354] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 4. Current term is 5.
I20260925 03:04:53.213608 24465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1e85248d3b04b979dd080caeeb02fdb" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:52.811194 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43366a0a2f9d4344bf4e37ffd290ef8a" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:52.892472 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be4e0aa984f4f2b81a58e382e434ad5" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:53.198822 24467 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 1097) took 1336 ms. Trace:
I20260925 03:04:52.809098 25198 raft_consensus.cc:2454] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 8.
I20260925 03:04:53.215175 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6b6ec010caf4689b8c90449764f5cf9" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:53.216535 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e5819c15eac4544a57e5ee6c9d4f117" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
W20260925 03:04:53.089090 24371 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:57656 (request call id 83) took 12085 ms (client timeout 15000 ms). Trace:
I20260925 03:04:52.809412 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6b6ec010caf4689b8c90449764f5cf9" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:53.163261 24544 raft_consensus.cc:1261] T 420163a6cdce412ebc21c25fe3835638 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260925 03:04:53.047250 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442d4a29b1d743599f4b6b4871e26668" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:53.216957 24467 rpcz_store.cc:276] 0925 03:04:51.861871 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:52.749762 (+887891us) service_pool.cc:227] Handling call
0925 03:04:53.002238 (+252476us) spinlock_profiling.cc:243] Waited 1.31 ms on lock 0x7b540025fc28. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f437c4478b1 00007f437c449434 00007f437b412437 00007f437b411bc8 00007f4374af9c04 00007f4374af9b92 00007f4374af9ac1 00007f4374af9a3c 00007f4374af8cf2 00007f43748fa94c 00007f43748f9d86
0925 03:04:53.198717 (+196479us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3012352}
I20260925 03:04:53.071385 24542 raft_consensus.cc:1261] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
W20260925 03:04:52.919938 26696 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 2.376s user 0.033s sys 0.013s
I20260925 03:04:53.229278 24470 raft_consensus.cc:2379] T 8e5819c15eac4544a57e5ee6c9d4f117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 6: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:04:52.810714 26530 raft_consensus.cc:3032] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 LEADER]: Stepping down as leader of term 6
I20260925 03:04:53.233851 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21c43af5e6844fdaacc8c625fa01fd8d" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:53.225428 24469 raft_consensus.cc:2454] T 43366a0a2f9d4344bf4e37ffd290ef8a P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 3.
I20260925 03:04:53.230636 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367277daf8d04778b3fa4ed94d71580d" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:53.234347 24511 leader_election.cc:302] T 550118bc1a0641ce9803bdbf43f2069f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:53.234685 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c81a1b3b616492ab6c964f45af36077" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:53.226423 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442d4a29b1d743599f4b6b4871e26668" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:53.238238 24547 raft_consensus.cc:3037] T 21c43af5e6844fdaacc8c625fa01fd8d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:53.241447 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5f9b503b0ab43999d582015a367441e" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:53.242262 24472 raft_consensus.cc:2379] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 7: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:04:53.242044 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c85f5d1d3f004cf5817bb594fd8e6469" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
W20260925 03:04:53.227073 26696 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 2.740s user 0.039s sys 0.038s
I20260925 03:04:53.206648 26682 leader_election.cc:288] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:53.244325 24467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367277daf8d04778b3fa4ed94d71580d" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:53.247524 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48ca6ccd98194cf2af7b5278d15bf4ea" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:53.236248 24465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:53.234711 24466 raft_consensus.cc:3037] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:53.242751 24585 leader_election.cc:302] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:53.233187 26766 consensus_queue.cc:1035] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.003s
I20260925 03:04:53.256081 24465 raft_consensus.cc:2379] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1bc1f2e62ab44af995165bd5116a9d43 in current term 7: Already voted for candidate eeab93abfdb141a895002ec9f312296c in this term.
I20260925 03:04:53.254383 24545 raft_consensus.cc:2454] T 442d4a29b1d743599f4b6b4871e26668 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 5.
W20260925 03:04:52.907892 25162 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 94818us
I20260925 03:04:53.253940 26640 consensus_queue.cc:1035] T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.032s
I20260925 03:04:53.260461 24465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:53.265051 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5278a60f97da4e10aecd50a8d9a9a43c" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:53.261194 24473 raft_consensus.cc:2454] T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 4.
I20260925 03:04:53.263321 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "550118bc1a0641ce9803bdbf43f2069f" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:53.266151 24469 raft_consensus.cc:2379] T 550118bc1a0641ce9803bdbf43f2069f P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 5: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:04:53.270270 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e725b82985746b39c0cc8f671c95353" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:53.270967 24472 raft_consensus.cc:2379] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 6: Already voted for candidate eeab93abfdb141a895002ec9f312296c in this term.
I20260925 03:04:53.264230 26552 raft_consensus.cc:2781] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260925 03:04:53.273485 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06f4a7eaf6534e4986c9fd35911038e4" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:53.234068 26530 raft_consensus.cc:729] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:04:53.275938 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4ebd2b043024a548005c8b6b00b0449" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:53.272069 26552 raft_consensus.cc:482] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:53.277362 26552 raft_consensus.cc:3037] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Advancing to term 9
I20260925 03:04:53.278256 26719 raft_consensus.cc:3032] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 LEADER]: Stepping down as leader of term 4
I20260925 03:04:53.278631 26719 raft_consensus.cc:729] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:04:53.279520 26719 consensus_queue.cc:260] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:53.281770 24465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c81a1b3b616492ab6c964f45af36077" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:53.261399 26241 raft_consensus.cc:2781] T 550118bc1a0641ce9803bdbf43f2069f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260925 03:04:53.285005 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f1fac028a144058b92331c0d9b44ebf" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:53.284292 24473 raft_consensus.cc:2454] T b4ebd2b043024a548005c8b6b00b0449 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 2.
I20260925 03:04:53.286180 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e725b82985746b39c0cc8f671c95353" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:53.286476 24511 leader_election.cc:302] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:53.281205 24471 raft_consensus.cc:2454] T c85f5d1d3f004cf5817bb594fd8e6469 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 4.
I20260925 03:04:53.289453 24368 catalog_manager.cc:5495] T 3d89396576884f7c8901dba54f783947 P a04a37fc4aa7460cbfdbeeb4e451e1f5 reported cstate change: term changed from 1 to 3, leader changed from 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1) to eeab93abfdb141a895002ec9f312296c (127.22.193.3). New cstate: current_term: 3 leader_uuid: "eeab93abfdb141a895002ec9f312296c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } }
I20260925 03:04:53.291136 24368 catalog_manager.cc:5495] T 43366a0a2f9d4344bf4e37ffd290ef8a P a04a37fc4aa7460cbfdbeeb4e451e1f5 reported cstate change: term changed from 1 to 2, leader changed from 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1) to a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2). New cstate: current_term: 2 leader_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: UNKNOWN } } }
I20260925 03:04:53.293154 26719 raft_consensus.cc:3037] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:53.298530 24467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceab6b0fccd54e909a64f66fe81a9a1f" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:53.285826 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c81a1b3b616492ab6c964f45af36077" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:53.299247 24467 raft_consensus.cc:2454] T ceab6b0fccd54e909a64f66fe81a9a1f P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 6.
I20260925 03:04:53.276247 26756 raft_consensus.cc:3037] T cc2df40db08649c9a75e90000b878559 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:53.300539 24472 raft_consensus.cc:2379] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 6: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:04:53.287019 24464 raft_consensus.cc:2362] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 6. Re-sending same reply.
I20260925 03:04:53.306649 24544 raft_consensus.cc:1261] T 8e5819c15eac4544a57e5ee6c9d4f117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260925 03:04:53.308307 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "420163a6cdce412ebc21c25fe3835638" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:53.303740 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15861cd28e354598a3c6bfc4337c34bb" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:53.309170 24470 raft_consensus.cc:2396] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for term 7 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260925 03:04:53.312317 24470 raft_consensus.cc:1226] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 LEADER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 6. Current term is 7. Ops: []
I20260925 03:04:53.313071 24470 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55560 (request call id 1297) took 2969 ms. Trace:
I20260925 03:04:53.313385 24470 rpcz_store.cc:276] 0925 03:04:50.343887 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:53.311731 (+2967844us) service_pool.cc:227] Handling call
0925 03:04:53.312116 (+ 385us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:53.312844 (+ 728us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:53.313001 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:53.314620 24470 raft_consensus.cc:1226] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 5. Current term is 6. Ops: []
I20260925 03:04:53.315189 24470 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1079) took 1814 ms. Trace:
I20260925 03:04:53.315501 24470 rpcz_store.cc:276] 0925 03:04:51.500476 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:53.314127 (+1813651us) service_pool.cc:227] Handling call
0925 03:04:53.314440 (+ 313us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:53.314959 (+ 519us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:53.315115 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:53.316648 24470 raft_consensus.cc:1226] T 4e5332a5d329493991dff994e70ea92c P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 5. Current term is 6. Ops: []
I20260925 03:04:53.321422 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5332a5d329493991dff994e70ea92c" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:53.324018 25198 raft_consensus.cc:1261] T 8ac466793d094325ace224708b2158ed P 1bc1f2e62ab44af995165bd5116a9d43 [term 5 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260925 03:04:53.287554 26241 raft_consensus.cc:482] T 550118bc1a0641ce9803bdbf43f2069f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:53.327814 24616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e54ca1757ee4e42956a9cebbeca4f62" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:53.328218 24470 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1081) took 1819 ms. Trace:
I20260925 03:04:53.328346 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b5df303cd6a4d59aca5c18bbbff24ed" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:53.328598 24616 raft_consensus.cc:3037] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Advancing to term 8
I20260925 03:04:53.325316 24547 raft_consensus.cc:2454] T 21c43af5e6844fdaacc8c625fa01fd8d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 4.
I20260925 03:04:53.330338 24614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e5819c15eac4544a57e5ee6c9d4f117" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:53.331493 24469 raft_consensus.cc:1226] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 4. Current term is 5. Ops: []
I20260925 03:04:53.328665 24470 rpcz_store.cc:276] 0925 03:04:51.501534 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:53.316152 (+1814618us) service_pool.cc:227] Handling call
0925 03:04:53.316443 (+ 291us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:53.320951 (+ 4508us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:53.321140 (+ 189us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:53.330371 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7aa47c48404465b94a36fd819fe3c7a" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:53.333385 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21c43af5e6844fdaacc8c625fa01fd8d" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:53.335569 25199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e54ca1757ee4e42956a9cebbeca4f62" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43"
I20260925 03:04:53.338416 24469 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55560 (request call id 1301) took 2942 ms. Trace:
I20260925 03:04:53.338815 24469 rpcz_store.cc:276] 0925 03:04:50.390015 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:53.325586 (+2935571us) service_pool.cc:227] Handling call
0925 03:04:53.331330 (+ 5744us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:53.331876 (+ 546us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:53.332064 (+ 188us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:53.327517 26646 raft_consensus.cc:2781] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260925 03:04:53.343815 24471 raft_consensus.cc:1226] T 293ef69caf514ac98510a68495f54eda P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Rejecting Update request from peer 1bc1f2e62ab44af995165bd5116a9d43 for earlier term 4. Current term is 5. Ops: []
I20260925 03:04:53.346756 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f1fac028a144058b92331c0d9b44ebf" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:53.347522 24471 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38920 (request call id 130) took 3346 ms. Trace:
I20260925 03:04:53.347729 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06f4a7eaf6534e4986c9fd35911038e4" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:53.340992 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48ca6ccd98194cf2af7b5278d15bf4ea" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:53.347082 24473 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38920 (request call id 129) took 3350 ms. Trace:
I20260925 03:04:53.348858 24471 rpcz_store.cc:276] 0925 03:04:50.000783 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:53.306433 (+3305650us) service_pool.cc:227] Handling call
0925 03:04:53.323632 (+ 17199us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:53.347281 (+ 23649us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:53.347474 (+ 193us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:53.349339 24473 rpcz_store.cc:276] 0925 03:04:49.991964 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:53.299541 (+3307577us) service_pool.cc:227] Handling call
0925 03:04:53.327855 (+ 28314us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:53.334885 (+ 7030us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:53.334911 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:53.342354 (+ 7443us) raft_consensus.cc:1631] Marking committed up to 5
0925 03:04:53.342389 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:53.342444 (+ 55us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:53.342535 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:53.350652 24464 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1082) took 1848 ms. Trace:
I20260925 03:04:53.351096 24464 rpcz_store.cc:276] 0925 03:04:51.501906 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:53.319935 (+1818029us) service_pool.cc:227] Handling call
0925 03:04:53.332672 (+ 12737us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:53.350235 (+ 17563us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:53.350270 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:53.350342 (+ 72us) raft_consensus.cc:1631] Marking committed up to 2
0925 03:04:53.350391 (+ 49us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:53.350461 (+ 70us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:53.350583 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:53.352843 24468 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1078) took 1900 ms. Trace:
I20260925 03:04:53.348793 24465 raft_consensus.cc:1226] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Rejecting Update request from peer 1bc1f2e62ab44af995165bd5116a9d43 for earlier term 7. Current term is 8. Ops: []
I20260925 03:04:53.354605 24467 raft_consensus.cc:1226] T 06f4a7eaf6534e4986c9fd35911038e4 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 LEADER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 5. Current term is 6. Ops: []
W20260925 03:04:53.345538 26712 consensus_meta.cc:287] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c: Time spent flushing consensus metadata: real 0.659s user 0.011s sys 0.000s
W20260925 03:04:53.219367 24371 rpcz_store.cc:269] 0925 03:04:41.003416 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:41.005587 (+ 2171us) service_pool.cc:227] Handling call
0925 03:04:53.088809 (+12083222us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0925 03:04:49.268653 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0925 03:04:49.413477 (+144824us) write_op.cc:432] Acquiring schema lock in shared mode
0925 03:04:49.413538 (+ 61us) write_op.cc:435] Acquired schema lock
0925 03:04:49.413555 (+ 17us) tablet.cc:660] Decoding operations
0925 03:04:49.413975 (+ 420us) write_op.cc:620] Acquiring the partition lock for write op
0925 03:04:49.414077 (+ 102us) write_op.cc:641] Partition lock acquired for write op
0925 03:04:49.414132 (+ 55us) tablet.cc:683] Acquiring locks for 2 operations
0925 03:04:49.414463 (+ 331us) tablet.cc:699] Row locks acquired
0925 03:04:49.414477 (+ 14us) write_op.cc:260] PREPARE: finished
0925 03:04:49.414635 (+ 158us) write_op.cc:270] Start()
0925 03:04:49.414743 (+ 108us) write_op.cc:276] Timestamp: P: 1790305489414585 usec, L: 0
0925 03:04:49.414763 (+ 20us) op_driver.cc:331] REPLICATION: starting
0925 03:04:49.423755 (+ 8992us) log.cc:842] Serialized 912 byte log entry
0925 03:04:49.812065 (+388310us) spinlock_profiling.cc:243] Waited 3.08 ms on lock 0x7b44000232cc. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f43733ecf11 00007f43734baa95 00007f43734bb4b2 00007f4376163511 00007f437615ee1a 00007f4376152bee 00007f4376151f36 00007f4376151b58 00007f437615b551 00007f437615b509 00007f437615b499
0925 03:04:50.492871 (+680806us) op_driver.cc:444] REPLICATION: finished
0925 03:04:50.934609 (+441738us) write_op.cc:301] APPLY: starting
0925 03:04:50.952904 (+ 18295us) tablet.cc:1365] starting BulkCheckPresence
0925 03:04:50.984781 (+ 31877us) tablet.cc:1368] finished BulkCheckPresence
0925 03:04:50.990458 (+ 5677us) tablet.cc:1370] starting ApplyRowOperation cycle
0925 03:04:51.275190 (+284732us) tablet.cc:1381] finished ApplyRowOperation cycle
0925 03:04:51.275423 (+ 233us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0925 03:04:51.275473 (+ 50us) write_op.cc:312] APPLY: finished
0925 03:04:51.275686 (+ 213us) log.cc:842] Serialized 28 byte log entry
0925 03:04:51.398102 (+122416us) write_op.cc:489] Releasing partition, row and schema locks
0925 03:04:51.399066 (+ 964us) write_op.cc:454] Released schema lock
0925 03:04:51.399122 (+ 56us) write_op.cc:341] FINISH: Updating metrics
0925 03:04:51.399428 (+ 306us) write_op.cc:489] Releasing partition, row and schema locks
0925 03:04:51.399456 (+ 28us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":4247673,"reported_tablets":10,"child_traces":[["op",{"apply.queue_time_us":282129,"apply.run_cpu_time_us":42180,"apply.run_wall_time_us":481824,"num_ops":2,"prepare.queue_time_us":85560,"prepare.run_cpu_time_us":56183,"prepare.run_wall_time_us":271539,"raft.queue_time_us":619894,"raft.run_cpu_time_us":11025,"raft.run_wall_time_us":234579,"replication_time_us":1071114,"spinlock_wait_cycles":7080192,"thread_start_us":612893,"threads_started":5,"wal-append.queue_time_us":412348,"wal-append.run_cpu_time_us":14194,"wal-append.run_wall_time_us":2315653}]]}
I20260925 03:04:53.354665 24468 rpcz_store.cc:276] 0925 03:04:51.499746 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:53.321036 (+1821290us) service_pool.cc:227] Handling call
0925 03:04:53.333126 (+ 12090us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:53.352374 (+ 19248us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:53.352415 (+ 41us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:53.352488 (+ 73us) raft_consensus.cc:1631] Marking committed up to 2
0925 03:04:53.352527 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:53.352601 (+ 74us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:53.352777 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:53.357753 24467 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55560 (request call id 1281) took 3105 ms. Trace:
I20260925 03:04:53.358250 24467 rpcz_store.cc:276] 0925 03:04:50.249966 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:53.309757 (+3059791us) service_pool.cc:227] Handling call
0925 03:04:53.344747 (+ 34990us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:53.355594 (+ 10847us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:53.355762 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:53.329851 26241 raft_consensus.cc:3037] T 550118bc1a0641ce9803bdbf43f2069f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:53.365806 24473 raft_consensus.cc:1226] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 6. Current term is 7. Ops: []
I20260925 03:04:53.366577 24473 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1089) took 1862 ms. Trace:
I20260925 03:04:53.366966 24473 rpcz_store.cc:276] 0925 03:04:51.529938 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:53.350260 (+1820322us) service_pool.cc:227] Handling call
0925 03:04:53.365642 (+ 15382us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:53.366316 (+ 674us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:53.366505 (+ 189us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:53.352555 26646 raft_consensus.cc:482] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:53.369628 26646 raft_consensus.cc:3037] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:04:53.333078 26530 consensus_queue.cc:260] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:53.369145 26552 raft_consensus.cc:504] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
W20260925 03:04:53.369226 26790 log.cc:924] Time spent T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.061s user 0.000s sys 0.000s
I20260925 03:04:53.376948 24615 raft_consensus.cc:1226] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 8. Current term is 9. Ops: []
I20260925 03:04:53.352597 26656 consensus_queue.cc:1035] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.033s
I20260925 03:04:53.381552 24471 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1088) took 1877 ms. Trace:
I20260925 03:04:53.382054 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be4e0aa984f4f2b81a58e382e434ad5" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:53.383170 24542 raft_consensus.cc:3037] T 0be4e0aa984f4f2b81a58e382e434ad5 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:53.382905 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5278a60f97da4e10aecd50a8d9a9a43c" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:53.384966 24539 raft_consensus.cc:3037] T 5278a60f97da4e10aecd50a8d9a9a43c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:53.383055 26764 raft_consensus.cc:3037] T 367277daf8d04778b3fa4ed94d71580d P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:53.389106 26552 leader_election.cc:288] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 9 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
W20260925 03:04:53.389951 26787 log.cc:924] Time spent T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.062s user 0.000s sys 0.000s
I20260925 03:04:53.391001 25199 raft_consensus.cc:1226] T 04e068a0f80941dc9138cee063f53bab P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 3. Current term is 4. Ops: []
I20260925 03:04:53.381983 24471 rpcz_store.cc:276] 0925 03:04:51.504294 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:53.349976 (+1845682us) service_pool.cc:227] Handling call
0925 03:04:53.381060 (+ 31084us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:53.381225 (+ 165us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:53.381248 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:53.381283 (+ 35us) raft_consensus.cc:1631] Marking committed up to 3
0925 03:04:53.381351 (+ 68us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:53.381401 (+ 50us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:53.381498 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:53.387663 24465 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38920 (request call id 128) took 3364 ms. Trace:
I20260925 03:04:53.394217 24465 rpcz_store.cc:276] 0925 03:04:49.989839 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:53.297004 (+3307165us) service_pool.cc:227] Handling call
0925 03:04:53.333471 (+ 36467us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:53.353943 (+ 20472us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:53.354166 (+ 223us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:53.394544 24472 raft_consensus.cc:1261] T 43366a0a2f9d4344bf4e37ffd290ef8a P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260925 03:04:53.397375 25163 leader_election.cc:302] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:04:53.401546 24472 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1085) took 1898 ms. Trace:
I20260925 03:04:53.401808 24472 rpcz_store.cc:276] 0925 03:04:51.503072 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:53.335982 (+1832910us) service_pool.cc:227] Handling call
0925 03:04:53.358875 (+ 22893us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:53.401348 (+ 42473us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:53.401499 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:53.407791 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c88cbd39e01f44628a094af516be2833" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:53.408360 24470 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1083) took 1906 ms. Trace:
I20260925 03:04:53.408248 24464 raft_consensus.cc:1226] T 1414dcfe53f64a78ae413531eb8ec117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 2. Current term is 3. Ops: []
I20260925 03:04:53.409531 24464 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1090) took 1869 ms. Trace:
I20260925 03:04:53.408773 24470 rpcz_store.cc:276] 0925 03:04:51.502334 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:53.334919 (+1832585us) service_pool.cc:227] Handling call
0925 03:04:53.355273 (+ 20354us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:53.408013 (+ 52740us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:53.408044 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:53.408087 (+ 43us) raft_consensus.cc:1631] Marking committed up to 2
0925 03:04:53.408119 (+ 32us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:53.408175 (+ 56us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:53.408303 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:53.409765 24468 raft_consensus.cc:1226] T 8e5819c15eac4544a57e5ee6c9d4f117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 5. Current term is 6. Ops: []
I20260925 03:04:53.411504 24468 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1091) took 1796 ms. Trace:
I20260925 03:04:53.410512 24464 rpcz_store.cc:276] 0925 03:04:51.543012 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:53.352169 (+1809157us) service_pool.cc:227] Handling call
0925 03:04:53.406294 (+ 54125us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:53.409278 (+ 2984us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:53.409473 (+ 195us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:53.411610 25199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5aec38e66a947c681e6c2a72b05e6ca" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43"
I20260925 03:04:53.414219 25199 raft_consensus.cc:3037] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43 [term 8 FOLLOWER]: Advancing to term 9
I20260925 03:04:53.415572 24471 raft_consensus.cc:1261] T ceab6b0fccd54e909a64f66fe81a9a1f P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260925 03:04:53.412058 24468 rpcz_store.cc:276] 0925 03:04:51.635277 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:53.358452 (+1723175us) service_pool.cc:227] Handling call
0925 03:04:53.407068 (+ 48616us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:53.411239 (+ 4171us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:53.411440 (+ 201us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:53.417508 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7691c6b475428c82d6181d693d26f3" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:53.418803 24542 raft_consensus.cc:2454] T 0be4e0aa984f4f2b81a58e382e434ad5 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 5.
I20260925 03:04:53.425112 24615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c85f5d1d3f004cf5817bb594fd8e6469" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:53.426296 24467 raft_consensus.cc:1226] T f22ddd7d1c444dc58f45a1fb0b3e02bc P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 5. Current term is 6. Ops: []
I20260925 03:04:53.395612 26530 raft_consensus.cc:3037] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:04:53.415162 24368 catalog_manager.cc:5495] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 reported cstate change: term changed from 3 to 4, leader changed from a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2) to eeab93abfdb141a895002ec9f312296c (127.22.193.3). New cstate: current_term: 4 leader_uuid: "eeab93abfdb141a895002ec9f312296c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } }
I20260925 03:04:53.430099 24465 raft_consensus.cc:1261] T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260925 03:04:53.430838 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b030d84c670b48d5bff337ccc0c77626" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:53.432405 24542 raft_consensus.cc:2454] T b030d84c670b48d5bff337ccc0c77626 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 6.
I20260925 03:04:53.441726 24613 raft_consensus.cc:1226] T 21c43af5e6844fdaacc8c625fa01fd8d P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 3. Current term is 4. Ops: []
I20260925 03:04:53.443638 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25102ba08e4e49388f6176badcd10a25" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:53.449777 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "420163a6cdce412ebc21c25fe3835638" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
W20260925 03:04:53.453325 26789 log.cc:924] Time spent T 0b5df303cd6a4d59aca5c18bbbff24ed P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.111s user 0.000s sys 0.001s
I20260925 03:04:53.454782 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c23337a7cb74184a0e57b7095cb6a11" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:53.456982 24615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "100c113a9fd84cc8adf2ec7b33ea0093" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:53.457562 24614 raft_consensus.cc:1226] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 6. Current term is 7. Ops: []
I20260925 03:04:53.463946 26646 raft_consensus.cc:504] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:53.466364 24586 leader_election.cc:302] T 442d4a29b1d743599f4b6b4871e26668 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:53.467298 24547 raft_consensus.cc:2379] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c in current term 7: Already voted for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in this term.
I20260925 03:04:53.468443 24539 raft_consensus.cc:2454] T 5278a60f97da4e10aecd50a8d9a9a43c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 4.
I20260925 03:04:53.470872 25199 raft_consensus.cc:2454] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 9.
I20260925 03:04:53.476526 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d89396576884f7c8901dba54f783947" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:53.478468 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ac285175464fbda67211d9574db415" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:53.475267 24615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abd5b8dc1b7e49e3aa9af900d73ff0e7" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:53.493069 24586 leader_election.cc:302] T 21c43af5e6844fdaacc8c625fa01fd8d P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:53.412715 26758 consensus_queue.cc:1046] T 04e068a0f80941dc9138cee063f53bab P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.038s
I20260925 03:04:53.466583 26646 leader_election.cc:288] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 7 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
W20260925 03:04:53.498342 24436 connection.cc:232] RPC call timeout handler was delayed by -1.386370476000593s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1595s for a response
W20260925 03:04:53.499985 24436 connection.cc:232] RPC call timeout handler was delayed by -1.3539704760005407s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16310000000000002s for a response
W20260925 03:04:53.500349 24436 connection.cc:232] RPC call timeout handler was delayed by -1.2396704759994464s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1758s for a response
I20260925 03:04:53.463896 24368 catalog_manager.cc:5495] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5 reported cstate change: term changed from 3 to 4, leader changed from eeab93abfdb141a895002ec9f312296c (127.22.193.3) to a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2). New cstate: current_term: 4 leader_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: HEALTHY } } }
I20260925 03:04:53.492758 26552 consensus_queue.cc:1035] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.010s
I20260925 03:04:53.495373 26378 raft_consensus.cc:2781] T 442d4a29b1d743599f4b6b4871e26668 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260925 03:04:53.499111 24614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9c33fd100ff44c1a3247319a324a044" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:53.499215 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceab6b0fccd54e909a64f66fe81a9a1f" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:53.507884 24615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5f9b503b0ab43999d582015a367441e" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:53.508545 24616 raft_consensus.cc:2454] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 8.
W20260925 03:04:53.501525 24511 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 215143us
W20260925 03:04:53.528280 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 104880us
I20260925 03:04:53.530619 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "550118bc1a0641ce9803bdbf43f2069f" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:53.531394 24619 raft_consensus.cc:2454] T 550118bc1a0641ce9803bdbf43f2069f P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 5.
I20260925 03:04:53.532889 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60d45b7d467d4e1ab92367f6d7d52b15" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
W20260925 03:04:53.536537 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 43567.8us
I20260925 03:04:53.542188 24547 raft_consensus.cc:1261] T 48ca6ccd98194cf2af7b5278d15bf4ea P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260925 03:04:53.507444 26450 consensus_meta.cc:287] T e92f377a79354a6ebb4eec67303c13e3 P 1bc1f2e62ab44af995165bd5116a9d43: Time spent flushing consensus metadata: real 0.706s user 0.008s sys 0.000s
I20260925 03:04:53.546938 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e54ca1757ee4e42956a9cebbeca4f62" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:53.550369 24468 raft_consensus.cc:2379] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c in current term 8: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:04:53.504905 26378 raft_consensus.cc:482] T 442d4a29b1d743599f4b6b4871e26668 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Starting leader election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:53.559453 24368 catalog_manager.cc:5495] T 8ac466793d094325ace224708b2158ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260925 03:04:53.528602 26640 raft_consensus.cc:482] T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:53.562544 24586 leader_election.cc:302] T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:53.492492 26738 consensus_queue.cc:1035] T 420163a6cdce412ebc21c25fe3835638 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.055s
I20260925 03:04:53.566205 26656 raft_consensus.cc:2781] T 21c43af5e6844fdaacc8c625fa01fd8d P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader election won for term 4
I20260925 03:04:53.495672 26699 consensus_queue.cc:1035] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.067s
I20260925 03:04:53.578979 24467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48ca6ccd98194cf2af7b5278d15bf4ea" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:53.582815 24467 raft_consensus.cc:2379] T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c in current term 4: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
W20260925 03:04:53.551566 25162 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 264415us
I20260925 03:04:53.566983 26378 raft_consensus.cc:3037] T 442d4a29b1d743599f4b6b4871e26668 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:04:53.593602 24586 leader_election.cc:302] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:53.600848 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06f4a7eaf6534e4986c9fd35911038e4" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
W20260925 03:04:53.603209 25163 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 205958us
I20260925 03:04:53.585003 26762 raft_consensus.cc:3032] T 04e068a0f80941dc9138cee063f53bab P eeab93abfdb141a895002ec9f312296c [term 3 LEADER]: Stepping down as leader of term 3
I20260925 03:04:53.549733 26450 raft_consensus.cc:504] T e92f377a79354a6ebb4eec67303c13e3 P 1bc1f2e62ab44af995165bd5116a9d43 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:53.609261 24586 leader_election.cc:302] T 5278a60f97da4e10aecd50a8d9a9a43c P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:53.599128 26646 raft_consensus.cc:482] T f22ddd7d1c444dc58f45a1fb0b3e02bc P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:53.612565 26803 raft_consensus.cc:2781] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260925 03:04:53.598572 26749 raft_consensus.cc:2781] T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Leader election won for term 5
I20260925 03:04:53.612939 24368 catalog_manager.cc:5495] T 21c43af5e6844fdaacc8c625fa01fd8d P a04a37fc4aa7460cbfdbeeb4e451e1f5 reported cstate change: term changed from 1 to 2, leader changed from 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1) to a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2). New cstate: current_term: 2 leader_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: UNKNOWN } } }
I20260925 03:04:53.614077 24467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:53.624907 24542 raft_consensus.cc:1261] T 8ac466793d094325ace224708b2158ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260925 03:04:53.629287 24547 raft_consensus.cc:1261] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260925 03:04:53.630158 24542 raft_consensus.cc:1226] T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 3. Current term is 4. Ops: []
I20260925 03:04:53.631266 24547 raft_consensus.cc:1261] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260925 03:04:53.629377 24467 raft_consensus.cc:2362] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 7. Re-sending same reply.
I20260925 03:04:53.608537 26656 raft_consensus.cc:686] T 21c43af5e6844fdaacc8c625fa01fd8d P eeab93abfdb141a895002ec9f312296c [term 4 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:53.610536 26762 raft_consensus.cc:729] T 04e068a0f80941dc9138cee063f53bab P eeab93abfdb141a895002ec9f312296c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:53.620769 26696 raft_consensus.cc:2781] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260925 03:04:53.640518 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be4e0aa984f4f2b81a58e382e434ad5" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
W20260925 03:04:53.642498 26793 log.cc:924] Time spent T 25102ba08e4e49388f6176badcd10a25 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.050s user 0.000s sys 0.001s
I20260925 03:04:53.636729 25162 leader_election.cc:302] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:53.625044 26552 raft_consensus.cc:482] T 71f461a6e2ff434993b78a2d077f0abd P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:53.628010 26803 raft_consensus.cc:482] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:53.592303 26640 raft_consensus.cc:504] T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:04:53.676066 25162 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 54276.6us
I20260925 03:04:53.665138 24511 leader_election.cc:302] T 367277daf8d04778b3fa4ed94d71580d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 1bc1f2e62ab44af995165bd5116a9d43, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:53.678197 24511 leader_election.cc:302] T b4ebd2b043024a548005c8b6b00b0449 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:04:53.636572 26756 raft_consensus.cc:2781] T 5278a60f97da4e10aecd50a8d9a9a43c P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader election won for term 4
I20260925 03:04:53.672744 25165 leader_election.cc:302] T b4ebd2b043024a548005c8b6b00b0449 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5; no voters:
I20260925 03:04:53.675633 24466 raft_consensus.cc:2454] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 5.
I20260925 03:04:53.679119 24511 leader_election.cc:302] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:04:53.673808 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5278a60f97da4e10aecd50a8d9a9a43c" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:53.639058 26749 raft_consensus.cc:686] T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c [term 5 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:53.643781 25194 raft_consensus.cc:1226] T e92f377a79354a6ebb4eec67303c13e3 P 1bc1f2e62ab44af995165bd5116a9d43 [term 5 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 4. Current term is 5. Ops: []
I20260925 03:04:53.644313 26696 raft_consensus.cc:482] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:53.643332 26697 consensus_queue.cc:1035] T 48ca6ccd98194cf2af7b5278d15bf4ea P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.038s
I20260925 03:04:53.639246 26682 consensus_queue.cc:1035] T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.045s
I20260925 03:04:53.639262 26646 raft_consensus.cc:504] T f22ddd7d1c444dc58f45a1fb0b3e02bc P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:53.719275 24511 leader_election.cc:302] T a4abe0ea757e4886be4945c5001e529a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 1bc1f2e62ab44af995165bd5116a9d43, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:53.722141 25162 leader_election.cc:302] T 7e49248643df4131a1c42e74244e05c2 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
W20260925 03:04:53.725101 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 43832.6us
I20260925 03:04:53.724627 24368 catalog_manager.cc:5495] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5 reported cstate change: term changed from 2 to 4, leader changed from 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1) to eeab93abfdb141a895002ec9f312296c (127.22.193.3). New cstate: current_term: 4 leader_uuid: "eeab93abfdb141a895002ec9f312296c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } }
I20260925 03:04:53.725976 24514 leader_election.cc:302] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:53.730247 24511 leader_election.cc:302] T 8ac466793d094325ace224708b2158ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:04:53.680188 26552 raft_consensus.cc:504] T 71f461a6e2ff434993b78a2d077f0abd P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:53.693534 26756 raft_consensus.cc:686] T 5278a60f97da4e10aecd50a8d9a9a43c P eeab93abfdb141a895002ec9f312296c [term 4 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:53.694438 26807 raft_consensus.cc:2726] T 367277daf8d04778b3fa4ed94d71580d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260925 03:04:53.680493 26803 raft_consensus.cc:3037] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:53.695420 26696 raft_consensus.cc:3037] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Advancing to term 8
I20260925 03:04:53.695434 26719 raft_consensus.cc:2781] T b4ebd2b043024a548005c8b6b00b0449 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260925 03:04:53.755034 24436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 118402us
I20260925 03:04:53.729822 26800 raft_consensus.cc:2781] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260925 03:04:53.756434 24465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b030d84c670b48d5bff337ccc0c77626" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:53.739089 26682 consensus_queue.cc:1225] T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Peer eeab93abfdb141a895002ec9f312296c log is divergent from this leader: its last log entry 3.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260925 03:04:53.765507 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7691c6b475428c82d6181d693d26f3" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:53.766955 24511 leader_election.cc:302] T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 1bc1f2e62ab44af995165bd5116a9d43, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:53.769339 24511 leader_election.cc:302] T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:04:53.771394 24465 raft_consensus.cc:2454] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 7.
I20260925 03:04:53.801393 24467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5aec38e66a947c681e6c2a72b05e6ca" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:53.771179 26807 raft_consensus.cc:2781] T 8ac466793d094325ace224708b2158ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260925 03:04:53.805943 26796 log.cc:924] Time spent T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.183s user 0.001s sys 0.000s
I20260925 03:04:53.720072 26762 consensus_queue.cc:260] T 04e068a0f80941dc9138cee063f53bab P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:53.756942 26719 raft_consensus.cc:482] T b4ebd2b043024a548005c8b6b00b0449 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:53.761392 26800 raft_consensus.cc:482] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:53.770098 26764 consensus_queue.cc:1035] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.041s
I20260925 03:04:53.801069 26741 consensus_queue.cc:1035] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260925 03:04:53.801118 26749 consensus_queue.cc:237] T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:53.802189 26810 raft_consensus.cc:2726] T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260925 03:04:53.813089 24511 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.4:43979, user_credentials={real_user=slave}} blocked reactor thread for 43816.5us
W20260925 03:04:53.904593 26798 log.cc:924] Time spent T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.166s user 0.006s sys 0.001s
I20260925 03:04:53.764106 26809 raft_consensus.cc:3037] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:53.756309 26811 raft_consensus.cc:2726] T a4abe0ea757e4886be4945c5001e529a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260925 03:04:53.801898 26804 raft_consensus.cc:2781] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260925 03:04:53.804816 26577 raft_consensus.cc:482] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:53.805220 26509 raft_consensus.cc:482] T 1414dcfe53f64a78ae413531eb8ec117 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:53.945544 26509 raft_consensus.cc:504] T 1414dcfe53f64a78ae413531eb8ec117 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:53.946465 26378 raft_consensus.cc:504] T 442d4a29b1d743599f4b6b4871e26668 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:53.947749 24621 raft_consensus.cc:1226] T 442d4a29b1d743599f4b6b4871e26668 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 5. Current term is 6. Ops: []
I20260925 03:04:53.948000 26509 leader_election.cc:288] T 1414dcfe53f64a78ae413531eb8ec117 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
W20260925 03:04:53.876727 25165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 340824us
W20260925 03:04:53.881454 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 111530us
W20260925 03:04:53.949986 26241 consensus_meta.cc:287] T 550118bc1a0641ce9803bdbf43f2069f P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent flushing consensus metadata: real 0.590s user 0.003s sys 0.008s
I20260925 03:04:53.898883 24368 catalog_manager.cc:5495] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 reported cstate change: term changed from 2 to 3, leader changed from 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) to a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2). New cstate: current_term: 3 leader_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260925 03:04:53.951090 24585 leader_election.cc:302] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, eeab93abfdb141a895002ec9f312296c; no voters:
W20260925 03:04:53.903522 25162 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 181459us
W20260925 03:04:53.904410 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 135757us
W20260925 03:04:53.884640 24436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 113502us
I20260925 03:04:53.876365 26450 leader_election.cc:288] T e92f377a79354a6ebb4eec67303c13e3 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 5 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:53.926546 24466 raft_consensus.cc:1261] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260925 03:04:53.756186 26656 consensus_queue.cc:237] T 21c43af5e6844fdaacc8c625fa01fd8d P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:53.928725 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e725b82985746b39c0cc8f671c95353" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:53.917694 26719 raft_consensus.cc:3037] T b4ebd2b043024a548005c8b6b00b0449 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:53.884622 26807 raft_consensus.cc:482] T 8ac466793d094325ace224708b2158ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:53.933128 24465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b627ecb4a5384ef392aa695e06599c39" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:53.942657 26804 raft_consensus.cc:482] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:53.883141 26799 raft_consensus.cc:2781] T b4ebd2b043024a548005c8b6b00b0449 P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260925 03:04:53.942764 26814 log.cc:924] Time spent T 7993f9c3e2a44f408bbf5d1e395e2306 P 869b0a8575b54d08ae2817fad5dd42d5: Append to log took a long time: real 0.060s user 0.000s sys 0.000s
I20260925 03:04:53.730564 26766 consensus_queue.cc:1046] T b627ecb4a5384ef392aa695e06599c39 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.042s
I20260925 03:04:53.918207 26659 raft_consensus.cc:482] T 37a157996e944db2906110d87ecb0c54 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:53.957710 25162 leader_election.cc:302] T 29637f5a4a504ce6ba29033ccd0cd716 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:53.958415 24514 leader_election.cc:302] T 3d89396576884f7c8901dba54f783947 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:04:53.959224 24471 raft_consensus.cc:2362] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 6. Re-sending same reply.
I20260925 03:04:53.883091 26816 raft_consensus.cc:2781] T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader election won for term 4
I20260925 03:04:53.930277 26722 raft_consensus.cc:482] T cc2df40db08649c9a75e90000b878559 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:53.961243 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5aec38e66a947c681e6c2a72b05e6ca" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:53.928299 26699 raft_consensus.cc:482] T edcf6e6a422d4d198d4fbb0f7d3d845d P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:53.918406 26800 raft_consensus.cc:3037] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:53.933650 24472 raft_consensus.cc:1226] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 6. Current term is 7. Ops: []
I20260925 03:04:53.963300 25198 raft_consensus.cc:1226] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43 [term 9 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 8. Current term is 9. Ops: [8.6-8.6]
I20260925 03:04:53.963634 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9357d144aa49789faaf31fe5e6bf89" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
W20260925 03:04:53.929646 24437 leader_election.cc:334] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389): Timed out: RequestConsensusVote RPC to 127.22.193.2:36389 timed out after 2.043s (SENT)
I20260925 03:04:53.970654 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43366a0a2f9d4344bf4e37ffd290ef8a" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:53.970880 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e725b82985746b39c0cc8f671c95353" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
W20260925 03:04:53.971366 26797 log.cc:924] Time spent T 7993f9c3e2a44f408bbf5d1e395e2306 P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.310s user 0.001s sys 0.000s
W20260925 03:04:53.972196 26815 log.cc:924] Time spent T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5: Append to log took a long time: real 0.053s user 0.000s sys 0.000s
I20260925 03:04:53.951503 26577 raft_consensus.cc:504] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:53.958184 26293 consensus_queue.cc:1035] T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.015s
I20260925 03:04:53.965054 25194 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49794 (request call id 222) took 1355 ms. Trace:
I20260925 03:04:53.973990 26754 raft_consensus.cc:2781] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Leader election won for term 9
I20260925 03:04:53.975385 24545 raft_consensus.cc:1226] T 0be4e0aa984f4f2b81a58e382e434ad5 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 4. Current term is 5. Ops: []
I20260925 03:04:53.976438 26552 leader_election.cc:288] T 71f461a6e2ff434993b78a2d077f0abd P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:53.976800 26450 raft_consensus.cc:2726] T 7e49248643df4131a1c42e74244e05c2 P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260925 03:04:53.977241 24368 catalog_manager.cc:5495] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "eeab93abfdb141a895002ec9f312296c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } }
I20260925 03:04:53.932464 26758 consensus_queue.cc:1046] T 04e068a0f80941dc9138cee063f53bab P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.047s
I20260925 03:04:53.971302 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f68804eb5a244d87bb395d50683c6012" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:53.950397 26241 raft_consensus.cc:504] T 550118bc1a0641ce9803bdbf43f2069f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:53.937595 26756 consensus_queue.cc:237] T 5278a60f97da4e10aecd50a8d9a9a43c P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:53.992240 24466 raft_consensus.cc:1226] T c88cbd39e01f44628a094af516be2833 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 5. Current term is 6. Ops: []
I20260925 03:04:54.052302 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367277daf8d04778b3fa4ed94d71580d" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:54.297055 24467 raft_consensus.cc:3032] T b627ecb4a5384ef392aa695e06599c39 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 LEADER]: Stepping down as leader of term 5
I20260925 03:04:54.302062 24540 raft_consensus.cc:2354] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 5. Current term is 6.
I20260925 03:04:54.302954 24543 raft_consensus.cc:2379] T 367277daf8d04778b3fa4ed94d71580d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 in current term 5: Already voted for candidate 1bc1f2e62ab44af995165bd5116a9d43 in this term.
I20260925 03:04:54.149930 24542 raft_consensus.cc:1226] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 6. Current term is 7. Ops: []
I20260925 03:04:53.966073 26799 raft_consensus.cc:482] T b4ebd2b043024a548005c8b6b00b0449 P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:54.304150 26799 raft_consensus.cc:3037] T b4ebd2b043024a548005c8b6b00b0449 P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:54.051853 24473 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1105) took 1128 ms. Trace:
I20260925 03:04:54.304981 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "293ef69caf514ac98510a68495f54eda" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:54.305624 24541 raft_consensus.cc:3037] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:54.307289 24464 raft_consensus.cc:2379] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c in current term 5: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:04:54.307155 24368 catalog_manager.cc:5495] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 reported cstate change: term changed from 2 to 3, leader changed from 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1) to eeab93abfdb141a895002ec9f312296c (127.22.193.3). New cstate: current_term: 3 leader_uuid: "eeab93abfdb141a895002ec9f312296c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } }
I20260925 03:04:54.308177 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be4e0aa984f4f2b81a58e382e434ad5" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
W20260925 03:04:54.075904 24588 leader_election.cc:334] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187): Timed out: RequestConsensusVote RPC to 127.22.193.1:42187 timed out after 1.692s (SENT)
I20260925 03:04:54.223042 26378 leader_election.cc:288] T 442d4a29b1d743599f4b6b4871e26668 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 6 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
W20260925 03:04:54.223812 26819 log.cc:924] Time spent T 43366a0a2f9d4344bf4e37ffd290ef8a P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.208s user 0.000s sys 0.000s
I20260925 03:04:54.065932 26646 leader_election.cc:288] T f22ddd7d1c444dc58f45a1fb0b3e02bc P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:54.311308 26699 raft_consensus.cc:504] T edcf6e6a422d4d198d4fbb0f7d3d845d P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:54.312413 26816 raft_consensus.cc:686] T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 LEADER]: Becoming Leader. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:04:54.188103 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5332a5d329493991dff994e70ea92c" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:54.313572 26816 consensus_queue.cc:237] T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
W20260925 03:04:54.229918 26814 log.cc:924] Time spent T 7993f9c3e2a44f408bbf5d1e395e2306 P 869b0a8575b54d08ae2817fad5dd42d5: Append to log took a long time: real 0.146s user 0.000s sys 0.000s
W20260925 03:04:54.314626 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 238781us
W20260925 03:04:54.311812 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 393948us
I20260925 03:04:54.302281 24467 raft_consensus.cc:729] T b627ecb4a5384ef392aa695e06599c39 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:04:54.310976 24546 raft_consensus.cc:1226] T 21c43af5e6844fdaacc8c625fa01fd8d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 3. Current term is 4. Ops: []
I20260925 03:04:54.260470 24469 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1086) took 2757 ms. Trace:
I20260925 03:04:54.304556 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abd5b8dc1b7e49e3aa9af900d73ff0e7" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:54.052049 24465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5278a60f97da4e10aecd50a8d9a9a43c" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
W20260925 03:04:54.317684 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 367054us
I20260925 03:04:54.314566 24473 rpcz_store.cc:276] 0925 03:04:52.890097 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:53.367738 (+477641us) service_pool.cc:227] Handling call
0925 03:04:53.368065 (+ 327us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:53.368260 (+ 195us) raft_consensus.cc:1520] Early marking committed up to index 4
0925 03:04:53.603034 (+234774us) log.cc:842] Serialized 14 byte log entry
0925 03:04:54.017998 (+414964us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:54.018052 (+ 54us) raft_consensus.cc:1631] Marking committed up to 4
0925 03:04:54.018102 (+ 50us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:54.018176 (+ 74us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:54.018292 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":40963,"prepare.queue_time_us":257880,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":41,"raft.queue_time_us":296640,"raft.run_cpu_time_us":214,"raft.run_wall_time_us":106159,"thread_start_us":617919,"threads_started":3,"wal-append.queue_time_us":160585}
W20260925 03:04:54.320416 25162 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 362863us
I20260925 03:04:54.018713 24436 leader_election.cc:398] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 election: Vote denied by peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105) with higher term. Message: Invalid argument: T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 5. Current term is 6.
I20260925 03:04:54.321152 24436 leader_election.cc:401] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20260925 03:04:54.305239 25194 rpcz_store.cc:276] 0925 03:04:52.605536 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:52.606122 (+ 586us) service_pool.cc:227] Handling call
0925 03:04:52.606466 (+ 344us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:53.722513 (+1116047us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:53.933049 (+210536us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b54005c08d4. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d0f51 00007f43764cf99e 00007f437b4112e9 00007f4374af7684 00007f4374af7612 00007f4374af7541 00007f4374af74bc 00007f4374af6772 00007f43748fa94c 00007f43748f9d86
0925 03:04:53.960749 (+ 27700us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2351375232}
I20260925 03:04:54.083657 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e8af2641e124f9589c5ccfdac785f63" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:54.304311 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1414dcfe53f64a78ae413531eb8ec117" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:54.314596 26577 leader_election.cc:288] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:54.325008 26722 raft_consensus.cc:504] T cc2df40db08649c9a75e90000b878559 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:54.327694 26659 raft_consensus.cc:504] T 37a157996e944db2906110d87ecb0c54 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:53.964468 26807 raft_consensus.cc:3037] T 8ac466793d094325ace224708b2158ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Advancing to term 6
W20260925 03:04:54.218499 26812 log.cc:924] Time spent T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.135s user 0.000s sys 0.001s
I20260925 03:04:54.318053 24469 rpcz_store.cc:276] 0925 03:04:51.503461 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:53.339591 (+1836130us) service_pool.cc:227] Handling call
0925 03:04:53.396871 (+ 57280us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:53.409527 (+ 12656us) raft_consensus.cc:1520] Early marking committed up to index 3
0925 03:04:53.540721 (+131194us) log.cc:842] Serialized 14 byte log entry
0925 03:04:53.878511 (+337790us) log.cc:842] Serialized 14 byte log entry
0925 03:04:54.244264 (+365753us) spinlock_profiling.cc:243] Waited 4.97 ms on lock 0x7b1800087520. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f437c4478b1 00007f437c454661 00007f437615f82c 00007f437358a9dc 00007f437c3c1a39 00007f43735a78bb 00007f43735a5cbf 00007f43735a5155 00007f43764c83d0 00007f43764da576 00007f43764eb0d7
0925 03:04:54.244399 (+ 135us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:54.244461 (+ 62us) raft_consensus.cc:1631] Marking committed up to 3
0925 03:04:54.260109 (+ 15648us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:54.260235 (+ 126us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:54.260397 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":37277,"prepare.queue_time_us":227437,"prepare.run_cpu_time_us":94,"prepare.run_wall_time_us":82,"raft.queue_time_us":418861,"raft.run_cpu_time_us":424,"raft.run_wall_time_us":70703,"spinlock_wait_cycles":11437952,"thread_start_us":447948,"threads_started":4,"wal-append.queue_time_us":190128}
I20260925 03:04:54.333025 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442d4a29b1d743599f4b6b4871e26668" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:54.332994 24467 consensus_queue.cc:260] T b627ecb4a5384ef392aa695e06599c39 P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:54.372798 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c88cbd39e01f44628a094af516be2833" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:53.966022 26804 raft_consensus.cc:3037] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Advancing to term 5
W20260925 03:04:54.346645 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 393513us
I20260925 03:04:54.335534 26659 leader_election.cc:288] T 37a157996e944db2906110d87ecb0c54 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:54.084981 26640 leader_election.cc:288] T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:54.328384 26820 raft_consensus.cc:3037] T 29637f5a4a504ce6ba29033ccd0cd716 P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:04:54.001793 26803 raft_consensus.cc:504] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:54.382963 26803 leader_election.cc:288] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 5 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:54.384627 25199 raft_consensus.cc:1226] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43 [term 5 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 4. Current term is 5. Ops: []
W20260925 03:04:54.365836 25165 leader_election.cc:334] T 367277daf8d04778b3fa4ed94d71580d P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187): Timed out: RequestConsensusVote RPC to 127.22.193.1:42187 timed out after 1.842s (SENT)
W20260925 03:04:54.391310 26827 log.cc:924] Time spent T 25102ba08e4e49388f6176badcd10a25 P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.058s user 0.001s sys 0.000s
I20260925 03:04:54.332271 26722 leader_election.cc:288] T cc2df40db08649c9a75e90000b878559 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:54.348803 26828 raft_consensus.cc:482] T 367277daf8d04778b3fa4ed94d71580d P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:54.372201 26646 raft_consensus.cc:2781] T 3d89396576884f7c8901dba54f783947 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260925 03:04:54.375687 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7aa47c48404465b94a36fd819fe3c7a" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
W20260925 03:04:54.342684 26530 consensus_meta.cc:287] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent flushing consensus metadata: real 0.884s user 0.011s sys 0.000s
I20260925 03:04:54.366935 26762 raft_consensus.cc:3037] T 04e068a0f80941dc9138cee063f53bab P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:54.338157 26741 consensus_queue.cc:1046] T 0b5df303cd6a4d59aca5c18bbbff24ed P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.052s
I20260925 03:04:54.331382 26754 raft_consensus.cc:686] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [term 9 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:54.364918 26378 consensus_queue.cc:1046] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.035s
I20260925 03:04:54.336355 26809 raft_consensus.cc:2726] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260925 03:04:54.364634 24437 leader_election.cc:302] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:54.404093 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9c33fd100ff44c1a3247319a324a044" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:54.329173 24472 raft_consensus.cc:2454] T 1414dcfe53f64a78ae413531eb8ec117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 3.
W20260925 03:04:54.353994 26696 consensus_meta.cc:287] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c: Time spent flushing consensus metadata: real 0.590s user 0.003s sys 0.006s
I20260925 03:04:54.398365 26738 consensus_queue.cc:1046] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260925 03:04:54.355734 26241 leader_election.cc:288] T 550118bc1a0641ce9803bdbf43f2069f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
W20260925 03:04:54.451297 24436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 471444us
I20260925 03:04:54.410936 26810 raft_consensus.cc:482] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:54.365602 26699 leader_election.cc:288] T edcf6e6a422d4d198d4fbb0f7d3d845d P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:54.437147 26646 raft_consensus.cc:482] T 3d89396576884f7c8901dba54f783947 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:54.478972 26646 raft_consensus.cc:3037] T 3d89396576884f7c8901dba54f783947 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:54.479106 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442d4a29b1d743599f4b6b4871e26668" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:54.480110 26509 raft_consensus.cc:3032] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c [term 5 LEADER]: Stepping down as leader of term 5
I20260925 03:04:54.480517 26509 raft_consensus.cc:729] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c [term 5 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:54.481389 26754 consensus_queue.cc:237] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:54.482143 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1414dcfe53f64a78ae413531eb8ec117" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:54.480412 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:54.489717 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71f461a6e2ff434993b78a2d077f0abd" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:54.490640 24540 raft_consensus.cc:2396] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1bc1f2e62ab44af995165bd5116a9d43 for term 7 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260925 03:04:54.449265 26722 raft_consensus.cc:482] T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:54.492751 24467 raft_consensus.cc:3037] T b627ecb4a5384ef392aa695e06599c39 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:04:54.492425 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f1fac028a144058b92331c0d9b44ebf" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:54.454272 26811 raft_consensus.cc:482] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:54.499655 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e5819c15eac4544a57e5ee6c9d4f117" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:54.482467 26738 raft_consensus.cc:3032] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [term 5 LEADER]: Stepping down as leader of term 5
W20260925 03:04:54.492163 24585 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.501s user 0.068s sys 0.047s
I20260925 03:04:54.440622 26758 consensus_queue.cc:1046] T a7aa47c48404465b94a36fd819fe3c7a P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.051s
I20260925 03:04:54.506016 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e92f377a79354a6ebb4eec67303c13e3" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:54.505347 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442d4a29b1d743599f4b6b4871e26668" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
W20260925 03:04:54.503729 24585 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.513s user 0.068s sys 0.047s
W20260925 03:04:54.508251 24585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.4:43979, user_credentials={real_user=slave}} blocked reactor thread for 557205us
W20260925 03:04:54.508855 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 144357us
I20260925 03:04:54.511698 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc2df40db08649c9a75e90000b878559" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:54.511960 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37a157996e944db2906110d87ecb0c54" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:54.512359 24470 raft_consensus.cc:2454] T cc2df40db08649c9a75e90000b878559 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 5.
I20260925 03:04:54.512908 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c81a1b3b616492ab6c964f45af36077" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:54.513301 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37a157996e944db2906110d87ecb0c54" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:54.488493 26821 raft_consensus.cc:2726] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260925 03:04:54.521297 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b8e0d260a884ecdb20f79de46c36b19" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:54.521492 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6b6ec010caf4689b8c90449764f5cf9" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:54.513579 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21c43af5e6844fdaacc8c625fa01fd8d" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:54.522966 24436 leader_election.cc:302] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: 1bc1f2e62ab44af995165bd5116a9d43, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:54.529466 24469 raft_consensus.cc:2454] T 7b8e0d260a884ecdb20f79de46c36b19 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 5.
I20260925 03:04:54.513645 24542 raft_consensus.cc:2454] T 37a157996e944db2906110d87ecb0c54 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 6.
I20260925 03:04:54.531436 24465 raft_consensus.cc:3032] T 9e8af2641e124f9589c5ccfdac785f63 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 LEADER]: Stepping down as leader of term 4
I20260925 03:04:54.531836 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367277daf8d04778b3fa4ed94d71580d" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:54.533003 24542 raft_consensus.cc:2362] T 367277daf8d04778b3fa4ed94d71580d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 5. Re-sending same reply.
I20260925 03:04:54.514838 26758 consensus_queue.cc:1225] T a7aa47c48404465b94a36fd819fe3c7a P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer 869b0a8575b54d08ae2817fad5dd42d5 log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260925 03:04:54.531785 24465 raft_consensus.cc:729] T 9e8af2641e124f9589c5ccfdac785f63 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:04:54.503389 26738 raft_consensus.cc:729] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [term 5 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:54.535218 24586 leader_election.cc:302] T 37a157996e944db2906110d87ecb0c54 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:54.536168 24616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e92f377a79354a6ebb4eec67303c13e3" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:54.476864 26696 raft_consensus.cc:504] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:54.532173 24473 raft_consensus.cc:2454] T 37a157996e944db2906110d87ecb0c54 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 6.
I20260925 03:04:54.537732 24615 raft_consensus.cc:1226] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Rejecting Update request from peer 1bc1f2e62ab44af995165bd5116a9d43 for earlier term 7. Current term is 8. Ops: []
I20260925 03:04:54.539592 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b8e0d260a884ecdb20f79de46c36b19" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:54.546289 24542 raft_consensus.cc:2454] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 4.
I20260925 03:04:54.480804 24471 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "21c43af5e6844fdaacc8c625fa01fd8d"
dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
mode: GRACEFUL
new_leader_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
from {username='slave'} at 127.0.0.1:38832
I20260925 03:04:54.548641 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1e85248d3b04b979dd080caeeb02fdb" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:54.549432 24621 raft_consensus.cc:2454] T b1e85248d3b04b979dd080caeeb02fdb P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 5.
I20260925 03:04:54.548868 24471 raft_consensus.cc:595] T 21c43af5e6844fdaacc8c625fa01fd8d P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 LEADER]: Received request to transfer leadership to TS a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:54.548862 24586 leader_election.cc:302] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:54.552368 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367277daf8d04778b3fa4ed94d71580d" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
W20260925 03:04:54.553874 25163 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 323189us
I20260925 03:04:54.554472 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5f9b503b0ab43999d582015a367441e" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:54.449175 26828 raft_consensus.cc:504] T 367277daf8d04778b3fa4ed94d71580d P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:54.481519 26509 consensus_queue.cc:260] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:54.553057 24465 consensus_queue.cc:260] T 9e8af2641e124f9589c5ccfdac785f63 P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:04:54.555718 25163 leader_election.cc:334] T 04e068a0f80941dc9138cee063f53bab P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389): Timed out: RequestConsensusVote RPC to 127.22.193.2:36389 timed out after 1.935s (SENT)
I20260925 03:04:54.558867 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e068a0f80941dc9138cee063f53bab" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:54.536947 26764 raft_consensus.cc:482] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:54.532547 26722 raft_consensus.cc:504] T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:54.530516 26699 consensus_queue.cc:1046] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.028s
I20260925 03:04:54.540212 26640 raft_consensus.cc:2726] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260925 03:04:54.567320 24541 raft_consensus.cc:2454] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 5.
I20260925 03:04:54.580826 24465 raft_consensus.cc:3037] T 9e8af2641e124f9589c5ccfdac785f63 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:54.581779 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7691c6b475428c82d6181d693d26f3" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:54.585458 25163 leader_election.cc:302] T 04e068a0f80941dc9138cee063f53bab P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:54.592120 24617 raft_consensus.cc:2454] T 367277daf8d04778b3fa4ed94d71580d P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 5.
I20260925 03:04:54.596673 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:54.597435 24541 raft_consensus.cc:2454] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 6.
W20260925 03:04:54.597432 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 188450us
I20260925 03:04:54.600656 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcf6e6a422d4d198d4fbb0f7d3d845d" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:54.601066 24467 raft_consensus.cc:1261] T b627ecb4a5384ef392aa695e06599c39 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260925 03:04:54.563493 26712 raft_consensus.cc:3032] T 0b5df303cd6a4d59aca5c18bbbff24ed P eeab93abfdb141a895002ec9f312296c [term 6 LEADER]: Stepping down as leader of term 6
W20260925 03:04:54.563400 26826 log.cc:924] Time spent T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.064s user 0.000s sys 0.000s
I20260925 03:04:54.555852 26756 raft_consensus.cc:2781] T 37a157996e944db2906110d87ecb0c54 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260925 03:04:54.601341 24542 raft_consensus.cc:2454] T edcf6e6a422d4d198d4fbb0f7d3d845d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 4.
I20260925 03:04:54.602658 26712 raft_consensus.cc:729] T 0b5df303cd6a4d59aca5c18bbbff24ed P eeab93abfdb141a895002ec9f312296c [term 6 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:54.566550 26293 raft_consensus.cc:3032] T a7aa47c48404465b94a36fd819fe3c7a P eeab93abfdb141a895002ec9f312296c [term 4 LEADER]: Stepping down as leader of term 4
I20260925 03:04:54.605851 26820 raft_consensus.cc:2726] T 29637f5a4a504ce6ba29033ccd0cd716 P 1bc1f2e62ab44af995165bd5116a9d43 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260925 03:04:54.552318 25194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71f461a6e2ff434993b78a2d077f0abd" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:54.503530 26811 raft_consensus.cc:504] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:54.558322 26656 raft_consensus.cc:2781] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260925 03:04:54.629136 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "420163a6cdce412ebc21c25fe3835638" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
W20260925 03:04:54.616489 25163 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 60747.3us
I20260925 03:04:54.618345 24586 leader_election.cc:302] T edcf6e6a422d4d198d4fbb0f7d3d845d P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
W20260925 03:04:54.621816 24514 leader_election.cc:334] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187): Timed out: RequestConsensusVote RPC to 127.22.193.1:42187 timed out after 1.610s (SENT)
I20260925 03:04:54.623961 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48ca6ccd98194cf2af7b5278d15bf4ea" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:54.563477 26825 raft_consensus.cc:2726] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105) with higher term. Message: Invalid argument: T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 5. Current term is 6.
W20260925 03:04:54.613920 26766 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 6 current term: 6 is higher.
I20260925 03:04:54.562840 26508 consensus_queue.cc:1046] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.066s
I20260925 03:04:54.567538 26552 consensus_queue.cc:1046] T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.058s
I20260925 03:04:54.576741 26828 leader_election.cc:288] T 367277daf8d04778b3fa4ed94d71580d P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:54.631673 24514 leader_election.cc:302] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:54.514173 26810 raft_consensus.cc:504] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:54.606287 26831 raft_consensus.cc:2726] T 04e068a0f80941dc9138cee063f53bab P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260925 03:04:54.618209 26293 raft_consensus.cc:729] T a7aa47c48404465b94a36fd819fe3c7a P eeab93abfdb141a895002ec9f312296c [term 4 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:54.641712 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367277daf8d04778b3fa4ed94d71580d" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
W20260925 03:04:54.581609 26682 consensus_queue.cc:1159] T 9e8af2641e124f9589c5ccfdac785f63 P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" responder_term: 4 status { last_received { term: 4 index: 4 } last_committed_idx: 4 last_received_current_leader { term: 4 index: 4 } }
I20260925 03:04:54.644122 24548 raft_consensus.cc:2396] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260925 03:04:54.623026 26756 raft_consensus.cc:482] T 37a157996e944db2906110d87ecb0c54 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:54.652426 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25102ba08e4e49388f6176badcd10a25" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:54.594303 26764 raft_consensus.cc:504] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:04:54.644968 24585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.4:43979, user_credentials={real_user=slave}} blocked reactor thread for 85119.5us
I20260925 03:04:54.570101 26823 consensus_queue.cc:1046] T 06f4a7eaf6534e4986c9fd35911038e4 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.061s
I20260925 03:04:54.648056 25198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "293ef69caf514ac98510a68495f54eda" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:54.647918 25199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71f461a6e2ff434993b78a2d077f0abd" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:54.649082 24542 raft_consensus.cc:2454] T 367277daf8d04778b3fa4ed94d71580d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 5.
I20260925 03:04:54.629266 26656 raft_consensus.cc:482] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260925 03:04:54.653411 24514 leader_election.cc:334] T cc2df40db08649c9a75e90000b878559 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187): Timed out: RequestConsensusVote RPC to 127.22.193.1:42187 timed out after 1.696s (SENT)
I20260925 03:04:54.630995 25194 raft_consensus.cc:2454] T 71f461a6e2ff434993b78a2d077f0abd P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 6.
I20260925 03:04:54.656210 26697 raft_consensus.cc:3032] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P eeab93abfdb141a895002ec9f312296c [term 4 LEADER]: Stepping down as leader of term 4
I20260925 03:04:54.657423 26682 consensus_queue.cc:1046] T 0be4e0aa984f4f2b81a58e382e434ad5 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260925 03:04:54.658349 26722 leader_election.cc:288] T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:54.562743 26738 consensus_queue.cc:260] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:54.661123 26766 consensus_queue.cc:1046] T 21c43af5e6844fdaacc8c625fa01fd8d P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260925 03:04:54.661706 26696 leader_election.cc:288] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 8 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:54.662603 26712 consensus_queue.cc:260] T 0b5df303cd6a4d59aca5c18bbbff24ed P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:54.714593 25199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:54.636334 26378 raft_consensus.cc:2781] T edcf6e6a422d4d198d4fbb0f7d3d845d P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260925 03:04:54.642859 26807 raft_consensus.cc:504] T 8ac466793d094325ace224708b2158ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:54.715452 26378 raft_consensus.cc:482] T edcf6e6a422d4d198d4fbb0f7d3d845d P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:54.715898 26378 raft_consensus.cc:3037] T edcf6e6a422d4d198d4fbb0f7d3d845d P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:54.715477 26738 raft_consensus.cc:3037] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:04:54.717108 26807 leader_election.cc:288] T 8ac466793d094325ace224708b2158ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 6 election: Requested vote from peers eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
W20260925 03:04:54.663936 26719 consensus_meta.cc:287] T b4ebd2b043024a548005c8b6b00b0449 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent flushing consensus metadata: real 0.665s user 0.005s sys 0.004s
I20260925 03:04:54.717864 26719 raft_consensus.cc:504] T b4ebd2b043024a548005c8b6b00b0449 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:54.720326 26719 leader_election.cc:288] T b4ebd2b043024a548005c8b6b00b0449 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:54.662401 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d89396576884f7c8901dba54f783947" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
W20260925 03:04:54.666733 24548 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36786 (request call id 976) took 1277 ms (client timeout 1696 ms). Trace:
I20260925 03:04:54.720971 24544 raft_consensus.cc:1226] T 8ac466793d094325ace224708b2158ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 5. Current term is 6. Ops: [5.7-5.7]
W20260925 03:04:54.721295 24548 rpcz_store.cc:269] 0925 03:04:53.389090 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:53.404212 (+ 15122us) service_pool.cc:227] Handling call
0925 03:04:54.666532 (+1262320us) spinlock_profiling.cc:243] Waited 1.22 s on lock 0x7b54003b8154. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d42ae 00007f437b411c27 00007f4374af9c04 00007f4374af9b92 00007f4374af9ac1 00007f4374af9a3c 00007f4374af8cf2 00007f43748fa94c 00007f43748f9d86 00007f43748f972f
0925 03:04:54.666672 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2810635648}
I20260925 03:04:54.721637 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b030d84c670b48d5bff337ccc0c77626" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:54.722203 24541 raft_consensus.cc:3037] T b030d84c670b48d5bff337ccc0c77626 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Advancing to term 8
I20260925 03:04:54.722518 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b5df303cd6a4d59aca5c18bbbff24ed" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:54.723666 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ac285175464fbda67211d9574db415" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:54.724258 24548 raft_consensus.cc:2454] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 5.
I20260925 03:04:54.672454 25162 leader_election.cc:302] T e92f377a79354a6ebb4eec67303c13e3 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:54.725052 25196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4ebd2b043024a548005c8b6b00b0449" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43"
W20260925 03:04:54.725287 25162 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 52949.7us
I20260925 03:04:54.723577 26811 leader_election.cc:288] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:54.726039 25162 leader_election.cc:302] T b1e85248d3b04b979dd080caeeb02fdb P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:04:54.689970 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4ebd2b043024a548005c8b6b00b0449" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:54.727383 26803 raft_consensus.cc:2726] T e92f377a79354a6ebb4eec67303c13e3 P 1bc1f2e62ab44af995165bd5116a9d43 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260925 03:04:54.727734 26820 raft_consensus.cc:2781] T b1e85248d3b04b979dd080caeeb02fdb P 1bc1f2e62ab44af995165bd5116a9d43 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260925 03:04:54.728046 26831 consensus_queue.cc:1046] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260925 03:04:54.664667 26293 consensus_queue.cc:260] T a7aa47c48404465b94a36fd819fe3c7a P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:54.644639 26804 raft_consensus.cc:504] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
W20260925 03:04:54.650694 26821 consensus_queue.cc:1159] T 9e8af2641e124f9589c5ccfdac785f63 P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "eeab93abfdb141a895002ec9f312296c" responder_term: 5 status { last_received { term: 4 index: 4 } last_committed_idx: 4 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 4. Current term is 5. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260925 03:04:54.728132 26820 raft_consensus.cc:482] T b1e85248d3b04b979dd080caeeb02fdb P 1bc1f2e62ab44af995165bd5116a9d43 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:54.705183 26823 consensus_queue.cc:1225] T 06f4a7eaf6534e4986c9fd35911038e4 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Peer a04a37fc4aa7460cbfdbeeb4e451e1f5 log is divergent from this leader: its last log entry 5.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260925 03:04:54.748659 26803 raft_consensus.cc:3032] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [term 7 LEADER]: Stepping down as leader of term 7
I20260925 03:04:54.728916 26828 raft_consensus.cc:3032] T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c [term 5 LEADER]: Stepping down as leader of term 5
I20260925 03:04:54.749253 26803 raft_consensus.cc:729] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 1bc1f2e62ab44af995165bd5116a9d43, State: Running, Role: LEADER
I20260925 03:04:54.750001 26824 raft_consensus.cc:3032] T 06f4a7eaf6534e4986c9fd35911038e4 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 LEADER]: Stepping down as leader of term 6
I20260925 03:04:54.746403 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c23337a7cb74184a0e57b7095cb6a11" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:54.709077 26697 raft_consensus.cc:729] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P eeab93abfdb141a895002ec9f312296c [term 4 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:54.750193 26803 consensus_queue.cc:260] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:54.727459 25194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ac466793d094325ace224708b2158ed" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43"
I20260925 03:04:54.751843 26803 raft_consensus.cc:3037] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [term 7 FOLLOWER]: Advancing to term 8
I20260925 03:04:54.702032 24471 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "37a157996e944db2906110d87ecb0c54"
dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
mode: GRACEFUL
new_leader_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
from {username='slave'} at 127.0.0.1:38832
I20260925 03:04:54.734025 26843 raft_consensus.cc:3032] T 21c43af5e6844fdaacc8c625fa01fd8d P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 LEADER]: Stepping down as leader of term 3
W20260925 03:04:54.733392 26834 log.cc:924] Time spent T 21c43af5e6844fdaacc8c625fa01fd8d P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.091s user 0.001s sys 0.000s
I20260925 03:04:54.704448 26656 raft_consensus.cc:3037] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:04:54.753345 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:54.732302 26509 raft_consensus.cc:3037] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:04:54.754084 26804 leader_election.cc:288] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 5 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:54.748978 25199 raft_consensus.cc:2454] T 7e49248643df4131a1c42e74244e05c2 P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 6.
I20260925 03:04:54.752480 24471 raft_consensus.cc:595] T 37a157996e944db2906110d87ecb0c54 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Received request to transfer leadership to TS a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:54.752539 26843 raft_consensus.cc:729] T 21c43af5e6844fdaacc8c625fa01fd8d P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:04:54.755095 24471 raft_consensus.cc:603] T 37a157996e944db2906110d87ecb0c54 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260925 03:04:54.750136 24542 raft_consensus.cc:2379] T b4ebd2b043024a548005c8b6b00b0449 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1bc1f2e62ab44af995165bd5116a9d43 in current term 4: Already voted for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in this term.
I20260925 03:04:54.741214 26818 consensus_queue.cc:1046] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.016s
I20260925 03:04:54.753981 24546 raft_consensus.cc:2454] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 6.
I20260925 03:04:54.749509 26828 raft_consensus.cc:729] T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c [term 5 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:54.715152 26712 raft_consensus.cc:3037] T 0b5df303cd6a4d59aca5c18bbbff24ed P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:04:54.748605 26820 raft_consensus.cc:3037] T b1e85248d3b04b979dd080caeeb02fdb P 1bc1f2e62ab44af995165bd5116a9d43 [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:04:54.758883 24471 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "420163a6cdce412ebc21c25fe3835638"
dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
mode: GRACEFUL
new_leader_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
from {username='slave'} at 127.0.0.1:38832
I20260925 03:04:54.759435 24471 raft_consensus.cc:595] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Received request to transfer leadership to TS a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:54.759792 24471 raft_consensus.cc:603] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260925 03:04:54.741447 25193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367277daf8d04778b3fa4ed94d71580d" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:54.741923 26764 leader_election.cc:288] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:54.760879 25193 raft_consensus.cc:2454] T 367277daf8d04778b3fa4ed94d71580d P 1bc1f2e62ab44af995165bd5116a9d43 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 5.
W20260925 03:04:54.717344 26833 log.cc:924] Time spent T 43366a0a2f9d4344bf4e37ffd290ef8a P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.069s user 0.000s sys 0.000s
I20260925 03:04:54.762274 24471 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "74443a0a1dc34d88a2f42e3b722db405"
dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
mode: GRACEFUL
new_leader_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
from {username='slave'} at 127.0.0.1:38832
I20260925 03:04:54.762874 24471 raft_consensus.cc:595] T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Received request to transfer leadership to TS a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:54.763852 26646 raft_consensus.cc:504] T 3d89396576884f7c8901dba54f783947 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:54.763197 24471 raft_consensus.cc:603] T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260925 03:04:54.758042 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceab6b0fccd54e909a64f66fe81a9a1f" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:54.772583 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "550118bc1a0641ce9803bdbf43f2069f" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:54.636209 26741 consensus_queue.cc:1046] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.054s
I20260925 03:04:54.667629 26756 raft_consensus.cc:3037] T 37a157996e944db2906110d87ecb0c54 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:04:54.713016 24585 leader_election.cc:302] T 71f461a6e2ff434993b78a2d077f0abd P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:54.714934 26682 raft_consensus.cc:3032] T 0be4e0aa984f4f2b81a58e382e434ad5 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 LEADER]: Stepping down as leader of term 4
I20260925 03:04:54.727459 25162 leader_election.cc:302] T 367277daf8d04778b3fa4ed94d71580d P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:04:54.788756 24465 raft_consensus.cc:1261] T 9e8af2641e124f9589c5ccfdac785f63 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260925 03:04:54.751435 26697 consensus_queue.cc:260] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:54.772871 26824 raft_consensus.cc:729] T 06f4a7eaf6534e4986c9fd35911038e4 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:04:54.750349 26799 raft_consensus.cc:504] T b4ebd2b043024a548005c8b6b00b0449 P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:54.738369 26816 raft_consensus.cc:2726] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260925 03:04:54.707015 24514 leader_election.cc:302] T cc2df40db08649c9a75e90000b878559 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:54.756417 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71f461a6e2ff434993b78a2d077f0abd" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:54.797693 24542 raft_consensus.cc:2454] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 6.
I20260925 03:04:54.757144 26818 consensus_queue.cc:1225] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Peer eeab93abfdb141a895002ec9f312296c log is divergent from this leader: its last log entry 7.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260925 03:04:54.772400 25194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 8 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:54.774408 24546 raft_consensus.cc:2454] T 550118bc1a0641ce9803bdbf43f2069f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 5.
W20260925 03:04:54.689749 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 51903.6us
I20260925 03:04:54.691210 24543 raft_consensus.cc:2379] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1bc1f2e62ab44af995165bd5116a9d43 in current term 7: Already voted for candidate eeab93abfdb141a895002ec9f312296c in this term.
I20260925 03:04:54.742898 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c85f5d1d3f004cf5817bb594fd8e6469" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:54.757937 26293 raft_consensus.cc:3037] T a7aa47c48404465b94a36fd819fe3c7a P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:54.783535 24436 leader_election.cc:302] T 550118bc1a0641ce9803bdbf43f2069f P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
W20260925 03:04:54.795085 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 188971us
I20260925 03:04:54.765393 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "100c113a9fd84cc8adf2ec7b33ea0093" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:54.746301 26749 raft_consensus.cc:3032] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [term 4 LEADER]: Stepping down as leader of term 4
I20260925 03:04:54.793689 26682 raft_consensus.cc:729] T 0be4e0aa984f4f2b81a58e382e434ad5 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
W20260925 03:04:54.818926 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 165546us
W20260925 03:04:54.819564 24514 leader_election.cc:334] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187): Timed out: RequestConsensusVote RPC to 127.22.193.1:42187 timed out after 1.714s (SENT)
I20260925 03:04:54.751537 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc2df40db08649c9a75e90000b878559" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
W20260925 03:04:54.799217 24585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.4:43979, user_credentials={real_user=slave}} blocked reactor thread for 86313.8us
I20260925 03:04:54.815864 24437 leader_election.cc:302] T 43366a0a2f9d4344bf4e37ffd290ef8a P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:54.802234 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f22ddd7d1c444dc58f45a1fb0b3e02bc" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
W20260925 03:04:54.803763 25162 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 76376.4us
I20260925 03:04:54.821835 24585 leader_election.cc:302] T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:54.805302 26799 leader_election.cc:288] T b4ebd2b043024a548005c8b6b00b0449 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 4 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:54.805526 26378 raft_consensus.cc:504] T edcf6e6a422d4d198d4fbb0f7d3d845d P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:54.830710 26378 leader_election.cc:288] T edcf6e6a422d4d198d4fbb0f7d3d845d P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:54.806130 26741 raft_consensus.cc:2781] T 71f461a6e2ff434993b78a2d077f0abd P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260925 03:04:54.806648 26764 raft_consensus.cc:3032] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c [term 5 LEADER]: Stepping down as leader of term 5
I20260925 03:04:54.831310 26741 raft_consensus.cc:482] T 71f461a6e2ff434993b78a2d077f0abd P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:54.831405 26764 raft_consensus.cc:729] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c [term 5 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:54.807338 25196 raft_consensus.cc:2379] T b4ebd2b043024a548005c8b6b00b0449 P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 4: Already voted for candidate 1bc1f2e62ab44af995165bd5116a9d43 in this term.
I20260925 03:04:54.764897 26844 consensus_queue.cc:1046] T 21c43af5e6844fdaacc8c625fa01fd8d P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.048s
I20260925 03:04:54.832816 24617 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9e8af2641e124f9589c5ccfdac785f63"
dest_uuid: "eeab93abfdb141a895002ec9f312296c"
mode: GRACEFUL
new_leader_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
from {username='slave'} at 127.0.0.1:44570
I20260925 03:04:54.833421 24617 raft_consensus.cc:595] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c [term 5 LEADER]: Received request to transfer leadership to TS a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:54.835795 26682 consensus_queue.cc:260] T 0be4e0aa984f4f2b81a58e382e434ad5 P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:54.837122 26682 raft_consensus.cc:3037] T 0be4e0aa984f4f2b81a58e382e434ad5 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:54.840525 26828 consensus_queue.cc:260] T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:54.816998 26749 raft_consensus.cc:729] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [term 4 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:54.768447 26800 raft_consensus.cc:504] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:54.813335 26824 consensus_queue.cc:260] T 06f4a7eaf6534e4986c9fd35911038e4 P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:54.799741 26804 raft_consensus.cc:2781] T 367277daf8d04778b3fa4ed94d71580d P 1bc1f2e62ab44af995165bd5116a9d43 [term 5 FOLLOWER]: Leader election won for term 5
I20260925 03:04:54.928954 26804 raft_consensus.cc:686] T 367277daf8d04778b3fa4ed94d71580d P 1bc1f2e62ab44af995165bd5116a9d43 [term 5 LEADER]: Becoming Leader. State: Replica: 1bc1f2e62ab44af995165bd5116a9d43, State: Running, Role: LEADER
I20260925 03:04:54.929900 26804 consensus_queue.cc:237] T 367277daf8d04778b3fa4ed94d71580d P 1bc1f2e62ab44af995165bd5116a9d43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 5, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:54.931679 26646 leader_election.cc:288] T 3d89396576884f7c8901dba54f783947 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:54.832472 26764 consensus_queue.cc:260] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:54.821445 26697 raft_consensus.cc:3037] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:54.918772 26821 raft_consensus.cc:3032] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 LEADER]: Stepping down as leader of term 8
I20260925 03:04:54.933427 26821 raft_consensus.cc:729] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:04:54.934839 26800 leader_election.cc:288] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:54.935559 24540 raft_consensus.cc:2379] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1bc1f2e62ab44af995165bd5116a9d43 in current term 5: Already voted for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in this term.
I20260925 03:04:54.935828 24543 raft_consensus.cc:1226] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 4. Current term is 5. Ops: []
I20260925 03:04:54.831774 26741 raft_consensus.cc:3037] T 71f461a6e2ff434993b78a2d077f0abd P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:04:54.932753 26656 raft_consensus.cc:504] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:54.941603 24547 raft_consensus.cc:1226] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 4. Current term is 5. Ops: []
W20260925 03:04:54.914476 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 77822.4us
W20260925 03:04:54.918610 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 98503.8us
I20260925 03:04:54.744009 26810 leader_election.cc:288] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:54.806604 26696 consensus_queue.cc:1046] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.102s
I20260925 03:04:54.870210 25194 raft_consensus.cc:2396] T 29637f5a4a504ce6ba29033ccd0cd716 P 1bc1f2e62ab44af995165bd5116a9d43 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for term 8 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20260925 03:04:54.813055 26816 raft_consensus.cc:2726] T cc2df40db08649c9a75e90000b878559 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260925 03:04:54.919858 25163 leader_election.cc:334] T e6b6ec010caf4689b8c90449764f5cf9 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389): Timed out: RequestConsensusVote RPC to 127.22.193.2:36389 timed out after 1.662s (SENT)
I20260925 03:04:54.796538 24541 raft_consensus.cc:2454] T b030d84c670b48d5bff337ccc0c77626 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 8.
I20260925 03:04:54.822801 24542 raft_consensus.cc:2454] T f22ddd7d1c444dc58f45a1fb0b3e02bc P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 6.
I20260925 03:04:54.926493 24546 raft_consensus.cc:1226] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 5. Current term is 6. Ops: []
I20260925 03:04:54.927541 26749 consensus_queue.cc:260] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:54.934377 26821 consensus_queue.cc:260] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:54.955425 26821 raft_consensus.cc:3037] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Advancing to term 9
W20260925 03:04:54.945694 24585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.4:43979, user_credentials={real_user=slave}} blocked reactor thread for 123957us
I20260925 03:04:54.959134 24585 leader_election.cc:302] T 367277daf8d04778b3fa4ed94d71580d P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:54.949894 26699 raft_consensus.cc:2781] T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260925 03:04:54.943032 24436 leader_election.cc:302] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:54.933265 26764 raft_consensus.cc:3037] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Advancing to term 6
W20260925 03:04:54.956045 25165 leader_election.cc:334] T e6b6ec010caf4689b8c90449764f5cf9 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187): Timed out: RequestConsensusVote RPC to 127.22.193.1:42187 timed out after 1.662s (SENT)
I20260925 03:04:54.956149 24539 raft_consensus.cc:2454] T cc2df40db08649c9a75e90000b878559 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 4.
I20260925 03:04:54.926713 26844 raft_consensus.cc:2726] T 43366a0a2f9d4344bf4e37ffd290ef8a P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260925 03:04:54.819478 26823 raft_consensus.cc:2781] T 550118bc1a0641ce9803bdbf43f2069f P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260925 03:04:54.958225 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15861cd28e354598a3c6bfc4337c34bb" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
W20260925 03:04:54.949280 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 133491us
I20260925 03:04:54.969107 24437 leader_election.cc:302] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:54.962715 25163 leader_election.cc:302] T e6b6ec010caf4689b8c90449764f5cf9 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
W20260925 03:04:54.969687 24436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 50485.1us
I20260925 03:04:54.962044 26699 raft_consensus.cc:482] T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Starting leader election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:54.973698 26699 raft_consensus.cc:3037] T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:04:54.964298 26846 consensus_queue.cc:1046] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260925 03:04:54.964793 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:54.976132 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71f461a6e2ff434993b78a2d077f0abd" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:54.978931 24547 raft_consensus.cc:2396] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1bc1f2e62ab44af995165bd5116a9d43 for term 7 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260925 03:04:54.982000 24542 raft_consensus.cc:1261] T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260925 03:04:54.968344 26823 raft_consensus.cc:482] T 550118bc1a0641ce9803bdbf43f2069f P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:54.985869 26823 raft_consensus.cc:3037] T 550118bc1a0641ce9803bdbf43f2069f P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:04:54.988828 26766 raft_consensus.cc:2781] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Leader election won for term 8
I20260925 03:04:54.991122 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c81a1b3b616492ab6c964f45af36077" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:54.991742 24617 raft_consensus.cc:3037] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:54.973803 26696 raft_consensus.cc:3032] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [term 5 LEADER]: Stepping down as leader of term 5
I20260925 03:04:54.993368 26696 raft_consensus.cc:729] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [term 5 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:54.994231 24548 raft_consensus.cc:2379] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1bc1f2e62ab44af995165bd5116a9d43 in current term 7: Already voted for candidate eeab93abfdb141a895002ec9f312296c in this term.
I20260925 03:04:54.995939 26378 consensus_queue.cc:1035] T 43366a0a2f9d4344bf4e37ffd290ef8a P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260925 03:04:54.997094 26846 consensus_queue.cc:1035] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260925 03:04:54.997493 26378 consensus_queue.cc:1035] T 48ca6ccd98194cf2af7b5278d15bf4ea P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260925 03:04:54.998446 24586 leader_election.cc:302] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:54.998950 26811 raft_consensus.cc:482] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:55.000108 26762 raft_consensus.cc:2781] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Leader election won for term 8
I20260925 03:04:55.000710 26762 raft_consensus.cc:686] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [term 8 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:55.000473 26811 raft_consensus.cc:504] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:55.001616 26762 consensus_queue.cc:237] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:55.002722 26811 leader_election.cc:288] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:54.996047 24588 leader_election.cc:302] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:04:54.997608 26758 raft_consensus.cc:3032] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c [term 6 LEADER]: Stepping down as leader of term 6
I20260925 03:04:55.003907 26758 raft_consensus.cc:729] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c [term 6 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:55.005394 24542 raft_consensus.cc:1226] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Rejecting Update request from peer 1bc1f2e62ab44af995165bd5116a9d43 for earlier term 4. Current term is 5. Ops: []
I20260925 03:04:55.005903 26293 raft_consensus.cc:2726] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260925 03:04:55.006492 26758 consensus_queue.cc:260] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:55.007445 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4ebd2b043024a548005c8b6b00b0449" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:55.008324 24546 raft_consensus.cc:2379] T b4ebd2b043024a548005c8b6b00b0449 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1bc1f2e62ab44af995165bd5116a9d43 in current term 4: Already voted for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in this term.
I20260925 03:04:55.010334 24588 leader_election.cc:302] T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:55.011533 26754 consensus_queue.cc:1046] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260925 03:04:55.014647 26758 raft_consensus.cc:3037] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:04:55.000154 24586 leader_election.cc:302] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:55.026685 26508 raft_consensus.cc:2781] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260925 03:04:55.027102 26508 raft_consensus.cc:482] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:55.027415 26508 raft_consensus.cc:3037] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:04:55.016968 26656 leader_election.cc:288] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 6 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:55.039651 26508 raft_consensus.cc:504] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:54.950254 26756 raft_consensus.cc:504] T 37a157996e944db2906110d87ecb0c54 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:55.047088 26756 leader_election.cc:288] T 37a157996e944db2906110d87ecb0c54 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 7 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:55.020334 26699 raft_consensus.cc:504] T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:55.017567 26820 raft_consensus.cc:504] T b1e85248d3b04b979dd080caeeb02fdb P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:54.978466 26712 raft_consensus.cc:2781] T 367277daf8d04778b3fa4ed94d71580d P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260925 03:04:55.042410 26508 leader_election.cc:288] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 7 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:55.044312 26293 raft_consensus.cc:3037] T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:04:55.044332 26824 raft_consensus.cc:3037] T 06f4a7eaf6534e4986c9fd35911038e4 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:04:55.048491 26803 raft_consensus.cc:2726] T e6b6ec010caf4689b8c90449764f5cf9 P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260925 03:04:55.050206 26749 raft_consensus.cc:3037] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Advancing to term 5
W20260925 03:04:55.051644 25165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 130112us
W20260925 03:04:55.048372 25163 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 128532us
I20260925 03:04:55.072613 25194 raft_consensus.cc:1261] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43 [term 9 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260925 03:04:55.073555 25163 leader_election.cc:302] T 71f461a6e2ff434993b78a2d077f0abd P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:55.067282 26825 consensus_queue.cc:1046] T 442d4a29b1d743599f4b6b4871e26668 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.007s
I20260925 03:04:55.074244 26754 consensus_queue.cc:1035] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.000s
I20260925 03:04:55.075322 26820 leader_election.cc:288] T b1e85248d3b04b979dd080caeeb02fdb P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 6 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:55.017519 24617 raft_consensus.cc:2454] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 5.
I20260925 03:04:55.075891 26820 raft_consensus.cc:2726] T 71f461a6e2ff434993b78a2d077f0abd P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260925 03:04:55.053386 24588 leader_election.cc:302] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters: 1bc1f2e62ab44af995165bd5116a9d43
I20260925 03:04:55.011924 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4ebd2b043024a548005c8b6b00b0449" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:55.055393 26828 raft_consensus.cc:3037] T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:04:55.015727 26823 raft_consensus.cc:504] T 550118bc1a0641ce9803bdbf43f2069f P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:55.051447 26818 raft_consensus.cc:2726] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260925 03:04:55.057318 26766 raft_consensus.cc:686] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 LEADER]: Becoming Leader. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:04:55.074368 26799 consensus_queue.cc:1046] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260925 03:04:55.077260 24541 raft_consensus.cc:2379] T b4ebd2b043024a548005c8b6b00b0449 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1bc1f2e62ab44af995165bd5116a9d43 in current term 4: Already voted for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in this term.
W20260925 03:04:55.058593 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 89571.7us
W20260925 03:04:55.056195 24436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 45821us
I20260925 03:04:55.077462 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1e85248d3b04b979dd080caeeb02fdb" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:55.089046 24617 raft_consensus.cc:3037] T b1e85248d3b04b979dd080caeeb02fdb P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:04:55.088340 24437 leader_election.cc:302] T c88cbd39e01f44628a094af516be2833 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:55.090178 24465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6b6ec010caf4689b8c90449764f5cf9" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:55.039081 26741 raft_consensus.cc:504] T 71f461a6e2ff434993b78a2d077f0abd P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:55.066766 26712 raft_consensus.cc:482] T 367277daf8d04778b3fa4ed94d71580d P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:55.084015 25196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6b6ec010caf4689b8c90449764f5cf9" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43"
I20260925 03:04:55.081013 25162 leader_election.cc:302] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:55.091527 24467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c81a1b3b616492ab6c964f45af36077" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
W20260925 03:04:55.096920 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 43628.6us
I20260925 03:04:55.098693 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1e85248d3b04b979dd080caeeb02fdb" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:55.099598 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15861cd28e354598a3c6bfc4337c34bb" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:55.101617 24542 raft_consensus.cc:1261] T 0be4e0aa984f4f2b81a58e382e434ad5 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260925 03:04:55.102949 24473 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b030d84c670b48d5bff337ccc0c77626"
dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
mode: GRACEFUL
new_leader_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
from {username='slave'} at 127.0.0.1:38832
I20260925 03:04:55.103510 24473 raft_consensus.cc:595] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Received request to transfer leadership to TS a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:55.103807 24473 raft_consensus.cc:603] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260925 03:04:55.104961 24465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4ebd2b043024a548005c8b6b00b0449" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:55.105422 24465 raft_consensus.cc:3037] T b4ebd2b043024a548005c8b6b00b0449 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Advancing to term 4
I20260925 03:04:55.090205 26640 raft_consensus.cc:3032] T 442d4a29b1d743599f4b6b4871e26668 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 LEADER]: Stepping down as leader of term 5
I20260925 03:04:55.107504 24437 leader_election.cc:302] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:55.110080 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcf6e6a422d4d198d4fbb0f7d3d845d" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:55.114486 24546 raft_consensus.cc:1261] T 21c43af5e6844fdaacc8c625fa01fd8d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260925 03:04:55.116129 24539 raft_consensus.cc:3037] T edcf6e6a422d4d198d4fbb0f7d3d845d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:55.117432 24617 raft_consensus.cc:2454] T b1e85248d3b04b979dd080caeeb02fdb P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 6.
I20260925 03:04:55.118561 25162 leader_election.cc:302] T b1e85248d3b04b979dd080caeeb02fdb P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:55.118664 23300 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260925 03:04:55.120672 23300 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260925 03:04:55.121232 26818 raft_consensus.cc:2726] T c88cbd39e01f44628a094af516be2833 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260925 03:04:55.121624 23300 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260925 03:04:55.128286 26293 raft_consensus.cc:2726] T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260925 03:04:55.097380 26508 raft_consensus.cc:2781] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Leader election won for term 7
I20260925 03:04:55.132985 26741 leader_election.cc:288] T 71f461a6e2ff434993b78a2d077f0abd P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 7 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:55.133100 26508 raft_consensus.cc:686] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [term 7 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:55.134028 26766 consensus_queue.cc:237] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:55.098737 26820 raft_consensus.cc:3032] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 LEADER]: Stepping down as leader of term 4
I20260925 03:04:55.136909 26820 raft_consensus.cc:729] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1bc1f2e62ab44af995165bd5116a9d43, State: Running, Role: LEADER
I20260925 03:04:55.137737 26820 consensus_queue.cc:260] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:55.138921 26820 raft_consensus.cc:3037] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:55.118731 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b8e0d260a884ecdb20f79de46c36b19" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:55.144572 24548 raft_consensus.cc:3037] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Advancing to term 6
I20260925 03:04:55.107087 25196 raft_consensus.cc:3037] T e6b6ec010caf4689b8c90449764f5cf9 P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:55.148922 26818 raft_consensus.cc:2726] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260925 03:04:55.109661 26799 raft_consensus.cc:2781] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43 [term 5 FOLLOWER]: Leader election won for term 5
I20260925 03:04:55.150799 26803 raft_consensus.cc:2781] T b1e85248d3b04b979dd080caeeb02fdb P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 FOLLOWER]: Leader election won for term 6
I20260925 03:04:55.099949 26712 raft_consensus.cc:3037] T 367277daf8d04778b3fa4ed94d71580d P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Advancing to term 6
W20260925 03:04:55.153960 24514 leader_election.cc:334] T 21c43af5e6844fdaacc8c625fa01fd8d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187): Timed out: RequestConsensusVote RPC to 127.22.193.1:42187 timed out after 1.548s (SENT)
I20260925 03:04:55.160181 24514 leader_election.cc:302] T 21c43af5e6844fdaacc8c625fa01fd8d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:55.153561 24437 leader_election.cc:302] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:55.160792 24541 raft_consensus.cc:1261] T 5278a60f97da4e10aecd50a8d9a9a43c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260925 03:04:55.156361 24465 raft_consensus.cc:2454] T b4ebd2b043024a548005c8b6b00b0449 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 4.
I20260925 03:04:55.116878 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37a157996e944db2906110d87ecb0c54" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:55.161834 24542 raft_consensus.cc:3037] T 37a157996e944db2906110d87ecb0c54 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:04:55.163187 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f22ddd7d1c444dc58f45a1fb0b3e02bc" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:55.163781 24546 raft_consensus.cc:3037] T f22ddd7d1c444dc58f45a1fb0b3e02bc P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:04:55.165694 24615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74443a0a1dc34d88a2f42e3b722db405" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:55.168475 24437 leader_election.cc:302] T 8e5819c15eac4544a57e5ee6c9d4f117 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:55.170646 24617 raft_consensus.cc:1226] T b1e85248d3b04b979dd080caeeb02fdb P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 5. Current term is 6. Ops: [2.5-5.7]
I20260925 03:04:55.086805 26656 consensus_queue.cc:1035] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.029s
I20260925 03:04:55.176131 24621 raft_consensus.cc:1226] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 4. Current term is 5. Ops: []
I20260925 03:04:55.174681 24548 raft_consensus.cc:2454] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 6.
I20260925 03:04:55.179060 24615 raft_consensus.cc:1226] T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 6. Current term is 7. Ops: []
I20260925 03:04:55.182046 24586 leader_election.cc:302] T cc2df40db08649c9a75e90000b878559 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:55.151422 26803 raft_consensus.cc:686] T b1e85248d3b04b979dd080caeeb02fdb P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 LEADER]: Becoming Leader. State: Replica: 1bc1f2e62ab44af995165bd5116a9d43, State: Running, Role: LEADER
I20260925 03:04:55.158744 26799 raft_consensus.cc:686] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43 [term 5 LEADER]: Becoming Leader. State: Replica: 1bc1f2e62ab44af995165bd5116a9d43, State: Running, Role: LEADER
I20260925 03:04:55.115612 26640 raft_consensus.cc:729] T 442d4a29b1d743599f4b6b4871e26668 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:04:55.182662 24616 raft_consensus.cc:1226] T 5278a60f97da4e10aecd50a8d9a9a43c P eeab93abfdb141a895002ec9f312296c [term 4 LEADER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 3. Current term is 4. Ops: []
I20260925 03:04:55.198201 24437 leader_election.cc:302] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
W20260925 03:04:55.185678 24514 leader_election.cc:334] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187): Timed out: RequestConsensusVote RPC to 127.22.193.1:42187 timed out after 1.637s (SENT)
I20260925 03:04:55.215634 24617 raft_consensus.cc:1261] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260925 03:04:55.213991 24539 raft_consensus.cc:2454] T edcf6e6a422d4d198d4fbb0f7d3d845d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 5.
I20260925 03:04:55.174226 26682 raft_consensus.cc:2726] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260925 03:04:55.220160 25194 raft_consensus.cc:1261] T 04e068a0f80941dc9138cee063f53bab P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260925 03:04:55.220783 26803 consensus_queue.cc:237] T b1e85248d3b04b979dd080caeeb02fdb P 1bc1f2e62ab44af995165bd5116a9d43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:55.221619 24621 raft_consensus.cc:1226] T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c [term 5 LEADER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 4. Current term is 5. Ops: []
I20260925 03:04:55.223675 24586 leader_election.cc:302] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:55.224203 26577 consensus_queue.cc:1035] T 21c43af5e6844fdaacc8c625fa01fd8d P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260925 03:04:55.224864 26378 consensus_queue.cc:1035] T 5278a60f97da4e10aecd50a8d9a9a43c P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260925 03:04:55.223562 26509 consensus_queue.cc:1035] T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260925 03:04:55.225940 26807 consensus_queue.cc:1035] T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20260925 03:04:55.226181 24514 leader_election.cc:334] T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187): Timed out: RequestConsensusVote RPC to 127.22.193.1:42187 timed out after 1.656s (SENT)
I20260925 03:04:55.231527 24514 leader_election.cc:302] T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:55.226734 24586 leader_election.cc:302] T edcf6e6a422d4d198d4fbb0f7d3d845d P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:55.216485 26818 raft_consensus.cc:2726] T 8e5819c15eac4544a57e5ee6c9d4f117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260925 03:04:55.225137 26741 raft_consensus.cc:2781] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Leader election won for term 6
I20260925 03:04:55.175902 26816 raft_consensus.cc:2726] T 21c43af5e6844fdaacc8c625fa01fd8d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260925 03:04:55.134097 26508 consensus_queue.cc:237] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:55.237914 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71f461a6e2ff434993b78a2d077f0abd" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:55.239339 25196 raft_consensus.cc:2454] T e6b6ec010caf4689b8c90449764f5cf9 P 1bc1f2e62ab44af995165bd5116a9d43 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 5.
I20260925 03:04:55.239122 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:55.299181 24542 raft_consensus.cc:2454] T 37a157996e944db2906110d87ecb0c54 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 7.
W20260925 03:04:55.304523 25165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 62769.1us
W20260925 03:04:55.239385 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 136267us
I20260925 03:04:55.316133 24588 leader_election.cc:302] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:55.313716 24511 leader_election.cc:302] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:04:55.312173 24539 raft_consensus.cc:3037] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:04:55.311827 24548 raft_consensus.cc:3037] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Advancing to term 7
W20260925 03:04:55.317960 26858 log.cc:924] Time spent T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.134s user 0.000s sys 0.000s
I20260925 03:04:55.261214 24621 raft_consensus.cc:1226] T 21c43af5e6844fdaacc8c625fa01fd8d P eeab93abfdb141a895002ec9f312296c [term 4 LEADER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 3. Current term is 4. Ops: []
W20260925 03:04:55.240025 24514 leader_election.cc:334] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187): Timed out: RequestConsensusVote RPC to 127.22.193.1:42187 timed out after 1.807s (SENT)
I20260925 03:04:55.321976 24514 leader_election.cc:302] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:55.214867 26738 raft_consensus.cc:2781] T cc2df40db08649c9a75e90000b878559 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260925 03:04:55.328619 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "550118bc1a0641ce9803bdbf43f2069f" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:55.236871 26807 raft_consensus.cc:2726] T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260925 03:04:55.243436 26830 log.cc:924] Time spent T 420163a6cdce412ebc21c25fe3835638 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.064s user 0.000s sys 0.000s
W20260925 03:04:55.334013 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 135916us
I20260925 03:04:55.336504 24621 raft_consensus.cc:3037] T 550118bc1a0641ce9803bdbf43f2069f P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:04:55.341701 25165 leader_election.cc:302] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:55.303251 26741 raft_consensus.cc:686] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [term 6 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:55.303241 26682 raft_consensus.cc:2726] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260925 03:04:55.348871 26803 raft_consensus.cc:2726] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260925 03:04:55.349488 26682 consensus_queue.cc:1046] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20260925 03:04:55.350333 26682 consensus_queue.cc:1046] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260925 03:04:55.353222 24437 leader_election.cc:302] T 293ef69caf514ac98510a68495f54eda P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 1bc1f2e62ab44af995165bd5116a9d43
I20260925 03:04:55.244393 26293 raft_consensus.cc:2781] T edcf6e6a422d4d198d4fbb0f7d3d845d P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Leader election won for term 5
I20260925 03:04:55.233925 26823 leader_election.cc:288] T 550118bc1a0641ce9803bdbf43f2069f P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:55.345300 25165 leader_election.cc:302] T b4ebd2b043024a548005c8b6b00b0449 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:55.314126 26699 leader_election.cc:288] T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 7 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:55.261087 25192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71f461a6e2ff434993b78a2d077f0abd" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43"
I20260925 03:04:55.364368 24621 raft_consensus.cc:2454] T 550118bc1a0641ce9803bdbf43f2069f P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 6.
I20260925 03:04:55.331421 26800 raft_consensus.cc:2781] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader election won for term 5
I20260925 03:04:55.356297 24548 raft_consensus.cc:2454] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 7.
I20260925 03:04:55.373040 24539 raft_consensus.cc:2454] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 7.
I20260925 03:04:55.376288 26766 raft_consensus.cc:3032] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 LEADER]: Stepping down as leader of term 6
I20260925 03:04:55.382323 25196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43"
I20260925 03:04:55.385171 24586 leader_election.cc:302] T 37a157996e944db2906110d87ecb0c54 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:55.340273 26738 raft_consensus.cc:482] T cc2df40db08649c9a75e90000b878559 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:55.349026 26577 raft_consensus.cc:2726] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260925 03:04:55.393261 26823 raft_consensus.cc:3032] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 LEADER]: Stepping down as leader of term 4
I20260925 03:04:55.385890 26766 raft_consensus.cc:729] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:04:55.336009 26818 consensus_queue.cc:1046] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.049s
I20260925 03:04:55.381979 26803 raft_consensus.cc:2781] T b4ebd2b043024a548005c8b6b00b0449 P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Leader election won for term 4
I20260925 03:04:55.386216 26719 raft_consensus.cc:2726] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260925 03:04:55.238394 26843 consensus_queue.cc:260] T 21c43af5e6844fdaacc8c625fa01fd8d P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:04:55.417060 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 183699us
W20260925 03:04:55.417768 24514 leader_election.cc:334] T 5278a60f97da4e10aecd50a8d9a9a43c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187): Timed out: RequestConsensusVote RPC to 127.22.193.1:42187 timed out after 1.832s (SENT)
I20260925 03:04:55.418151 24514 leader_election.cc:302] T 5278a60f97da4e10aecd50a8d9a9a43c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:55.403147 26878 consensus_queue.cc:1035] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.016s
W20260925 03:04:55.419929 24514 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.22.193.1:42187 timed out after 2.863s (SENT) [suppressed 89 similar messages]
I20260925 03:04:55.420864 26821 consensus_queue.cc:1046] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260925 03:04:55.421701 26821 consensus_queue.cc:1225] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Peer eeab93abfdb141a895002ec9f312296c log is divergent from this leader: its last log entry 4.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260925 03:04:55.422194 26821 raft_consensus.cc:482] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:55.422640 26821 raft_consensus.cc:504] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:55.394308 26823 raft_consensus.cc:729] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:04:55.425760 26823 consensus_queue.cc:260] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:55.426855 26823 raft_consensus.cc:3037] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:55.382529 26799 consensus_queue.cc:237] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:55.385793 25192 raft_consensus.cc:3037] T 71f461a6e2ff434993b78a2d077f0abd P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:04:55.399505 26293 raft_consensus.cc:686] T edcf6e6a422d4d198d4fbb0f7d3d845d P eeab93abfdb141a895002ec9f312296c [term 5 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:55.383688 26872 consensus_queue.cc:1046] T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.050s
I20260925 03:04:55.403120 26766 consensus_queue.cc:260] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:55.408862 26577 raft_consensus.cc:2781] T 37a157996e944db2906110d87ecb0c54 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Leader election won for term 7
I20260925 03:04:55.434185 26872 consensus_queue.cc:1225] T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Peer eeab93abfdb141a895002ec9f312296c log is divergent from this leader: its last log entry 3.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260925 03:04:55.434670 26577 raft_consensus.cc:686] T 37a157996e944db2906110d87ecb0c54 P eeab93abfdb141a895002ec9f312296c [term 7 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:55.435489 26577 consensus_queue.cc:237] T 37a157996e944db2906110d87ecb0c54 P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:55.409752 26552 consensus_queue.cc:1046] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260925 03:04:55.437521 26552 raft_consensus.cc:3032] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [term 6 LEADER]: Stepping down as leader of term 6
I20260925 03:04:55.437842 26552 raft_consensus.cc:729] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [term 6 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:55.438632 26552 consensus_queue.cc:260] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:55.439895 26552 raft_consensus.cc:3037] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:04:55.434127 26293 consensus_queue.cc:237] T edcf6e6a422d4d198d4fbb0f7d3d845d P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:55.388077 26738 raft_consensus.cc:3037] T cc2df40db08649c9a75e90000b878559 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:04:55.445992 24546 raft_consensus.cc:2454] T f22ddd7d1c444dc58f45a1fb0b3e02bc P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 7.
I20260925 03:04:55.401124 24588 leader_election.cc:302] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
W20260925 03:04:55.444746 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 44212.9us
I20260925 03:04:55.449163 24586 leader_election.cc:302] T 71f461a6e2ff434993b78a2d077f0abd P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
W20260925 03:04:55.452343 24514 leader_election.cc:334] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187): Timed out: RequestConsensusVote RPC to 127.22.193.1:42187 timed out after 1.932s (SENT)
I20260925 03:04:55.452824 24514 leader_election.cc:302] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:55.456813 24586 leader_election.cc:302] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:55.394078 26800 raft_consensus.cc:686] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 LEADER]: Becoming Leader. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:04:55.433233 26882 raft_consensus.cc:2781] T 293ef69caf514ac98510a68495f54eda P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader election won for term 5
I20260925 03:04:55.462678 24436 leader_election.cc:302] T 550118bc1a0641ce9803bdbf43f2069f P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:55.387905 26749 consensus_queue.cc:1035] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.060s
I20260925 03:04:55.418018 26803 raft_consensus.cc:686] T b4ebd2b043024a548005c8b6b00b0449 P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 LEADER]: Becoming Leader. State: Replica: 1bc1f2e62ab44af995165bd5116a9d43, State: Running, Role: LEADER
W20260925 03:04:55.467733 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 66695.9us
W20260925 03:04:55.468218 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 115074us
I20260925 03:04:55.468169 24539 raft_consensus.cc:1226] T 8ac466793d094325ace224708b2158ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260925 03:04:55.514930 24437 leader_election.cc:302] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
W20260925 03:04:55.443760 26881 consensus_queue.cc:1159] T 21c43af5e6844fdaacc8c625fa01fd8d P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "eeab93abfdb141a895002ec9f312296c" responder_term: 4 status { last_received { term: 4 index: 3 } last_committed_idx: 2 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 3. Current term is 4. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260925 03:04:55.492918 26883 raft_consensus.cc:3032] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 LEADER]: Stepping down as leader of term 3
I20260925 03:04:55.493527 26818 raft_consensus.cc:3032] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 LEADER]: Stepping down as leader of term 4
I20260925 03:04:55.489001 26800 consensus_queue.cc:237] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:55.459933 26764 raft_consensus.cc:2726] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260925 03:04:55.353214 26741 consensus_queue.cc:237] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:55.471500 26656 raft_consensus.cc:2781] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Leader election won for term 7
I20260925 03:04:55.467553 26378 raft_consensus.cc:2781] T 71f461a6e2ff434993b78a2d077f0abd P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Leader election won for term 7
I20260925 03:04:55.385142 26640 consensus_queue.cc:260] T 442d4a29b1d743599f4b6b4871e26668 P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:55.467568 26811 raft_consensus.cc:2726] T 5278a60f97da4e10aecd50a8d9a9a43c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260925 03:04:55.575058 26659 consensus_queue.cc:1046] T c88cbd39e01f44628a094af516be2833 P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.006s
W20260925 03:04:55.577538 26829 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.554s user 0.007s sys 0.000s
W20260925 03:04:55.581279 26829 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.558s user 0.007s sys 0.000s
I20260925 03:04:55.489881 26882 raft_consensus.cc:686] T 293ef69caf514ac98510a68495f54eda P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 LEADER]: Becoming Leader. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:04:55.582379 26882 consensus_queue.cc:237] T 293ef69caf514ac98510a68495f54eda P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:55.471558 26719 raft_consensus.cc:2726] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260925 03:04:55.584914 26640 raft_consensus.cc:3037] T 442d4a29b1d743599f4b6b4871e26668 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Advancing to term 6
W20260925 03:04:55.496091 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 43786.7us
I20260925 03:04:55.492492 26696 consensus_queue.cc:260] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:55.528139 26712 raft_consensus.cc:504] T 367277daf8d04778b3fa4ed94d71580d P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
W20260925 03:04:55.589195 24514 leader_election.cc:334] T ac9357d144aa49789faaf31fe5e6bf89 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187): Timed out: RequestConsensusVote RPC to 127.22.193.1:42187 timed out after 1.939s (SENT)
I20260925 03:04:55.589660 24514 leader_election.cc:302] T ac9357d144aa49789faaf31fe5e6bf89 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:55.590430 26719 raft_consensus.cc:2726] T ac9357d144aa49789faaf31fe5e6bf89 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260925 03:04:55.590777 24514 leader_election.cc:334] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187): Timed out: RequestConsensusVote RPC to 127.22.193.1:42187 timed out after 1.966s (SENT)
I20260925 03:04:55.591230 24514 leader_election.cc:302] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5
W20260925 03:04:55.451321 26874 consensus_queue.cc:1159] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "1bc1f2e62ab44af995165bd5116a9d43" responder_term: 6 status { last_received { term: 6 index: 5 } last_committed_idx: 4 last_received_current_leader { term: 6 index: 5 } }
I20260925 03:04:55.591399 26712 leader_election.cc:288] T 367277daf8d04778b3fa4ed94d71580d P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 6 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:55.519644 24616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:55.520844 24437 leader_election.cc:302] T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:55.522795 24615 raft_consensus.cc:1226] T 5278a60f97da4e10aecd50a8d9a9a43c P eeab93abfdb141a895002ec9f312296c [term 4 LEADER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 3. Current term is 4. Ops: []
I20260925 03:04:55.593060 24619 raft_consensus.cc:1226] T 367277daf8d04778b3fa4ed94d71580d P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Rejecting Update request from peer 1bc1f2e62ab44af995165bd5116a9d43 for earlier term 5. Current term is 6. Ops: []
I20260925 03:04:55.516999 26821 leader_election.cc:288] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:55.517386 26844 raft_consensus.cc:2781] T 550118bc1a0641ce9803bdbf43f2069f P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader election won for term 6
I20260925 03:04:55.523566 25194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:55.529230 26803 consensus_queue.cc:237] T b4ebd2b043024a548005c8b6b00b0449 P 1bc1f2e62ab44af995165bd5116a9d43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:55.476923 25192 raft_consensus.cc:2454] T 71f461a6e2ff434993b78a2d077f0abd P 1bc1f2e62ab44af995165bd5116a9d43 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 7.
I20260925 03:04:55.519302 26883 raft_consensus.cc:729] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
W20260925 03:04:55.602988 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 82235.7us
I20260925 03:04:55.603879 24437 leader_election.cc:302] T 25102ba08e4e49388f6176badcd10a25 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:55.596206 24588 leader_election.cc:302] T 1414dcfe53f64a78ae413531eb8ec117 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:55.565452 26818 raft_consensus.cc:729] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:04:55.567024 26682 raft_consensus.cc:2726] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260925 03:04:55.616326 24437 leader_election.cc:302] T 3d89396576884f7c8901dba54f783947 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:55.597261 26378 raft_consensus.cc:686] T 71f461a6e2ff434993b78a2d077f0abd P eeab93abfdb141a895002ec9f312296c [term 7 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:55.619885 24437 leader_election.cc:302] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:55.606350 25193 raft_consensus.cc:1226] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43 [term 5 LEADER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 4. Current term is 5. Ops: []
I20260925 03:04:55.605784 26823 raft_consensus.cc:2726] T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260925 03:04:55.605995 26719 raft_consensus.cc:2726] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260925 03:04:55.627902 24465 raft_consensus.cc:1261] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Refusing update from remote peer 1bc1f2e62ab44af995165bd5116a9d43: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260925 03:04:55.606518 26844 raft_consensus.cc:686] T 550118bc1a0641ce9803bdbf43f2069f P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 LEADER]: Becoming Leader. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:04:55.610422 26696 raft_consensus.cc:3037] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:04:55.609700 26656 raft_consensus.cc:686] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c [term 7 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:55.637068 24588 leader_election.cc:302] T 442d4a29b1d743599f4b6b4871e26668 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:55.608086 25194 raft_consensus.cc:2454] T 29637f5a4a504ce6ba29033ccd0cd716 P 1bc1f2e62ab44af995165bd5116a9d43 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 7.
I20260925 03:04:55.640026 24540 raft_consensus.cc:1261] T 367277daf8d04778b3fa4ed94d71580d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Refusing update from remote peer 1bc1f2e62ab44af995165bd5116a9d43: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260925 03:04:55.600530 26738 raft_consensus.cc:504] T cc2df40db08649c9a75e90000b878559 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:55.639403 26799 consensus_queue.cc:1035] T 367277daf8d04778b3fa4ed94d71580d P 1bc1f2e62ab44af995165bd5116a9d43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.004s
I20260925 03:04:55.641652 26754 raft_consensus.cc:3032] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c [term 5 LEADER]: Stepping down as leader of term 5
I20260925 03:04:55.641929 26846 raft_consensus.cc:2781] T 1414dcfe53f64a78ae413531eb8ec117 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260925 03:04:55.627312 26682 raft_consensus.cc:2726] T 3d89396576884f7c8901dba54f783947 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260925 03:04:55.644572 24615 raft_consensus.cc:1261] T b1e85248d3b04b979dd080caeeb02fdb P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Refusing update from remote peer 1bc1f2e62ab44af995165bd5116a9d43: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
W20260925 03:04:55.601715 26864 consensus_queue.cc:1159] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" responder_term: 6 status { last_received { term: 4 index: 6 } last_committed_idx: 6 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 5. Current term is 6. Ops: [5.7-5.7]" } } last_received_current_leader { term: 0 index: 0 } }
I20260925 03:04:55.621310 26821 raft_consensus.cc:2726] T 25102ba08e4e49388f6176badcd10a25 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260925 03:04:55.657825 24437 leader_election.cc:302] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:55.654747 26846 raft_consensus.cc:482] T 1414dcfe53f64a78ae413531eb8ec117 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260925 03:04:55.633795 26762 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.628s user 0.002s sys 0.012s
I20260925 03:04:55.646862 24588 leader_election.cc:302] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:04:55.659361 26846 raft_consensus.cc:3037] T 1414dcfe53f64a78ae413531eb8ec117 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:55.671396 26863 raft_consensus.cc:2726] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260925 03:04:55.661314 26656 consensus_queue.cc:237] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:55.651639 26883 consensus_queue.cc:260] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:55.673852 26883 raft_consensus.cc:3037] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:55.674727 26766 raft_consensus.cc:3037] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:04:55.665163 26804 consensus_queue.cc:1035] T 367277daf8d04778b3fa4ed94d71580d P 1bc1f2e62ab44af995165bd5116a9d43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260925 03:04:55.651355 26756 raft_consensus.cc:2726] T 442d4a29b1d743599f4b6b4871e26668 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260925 03:04:55.651950 26844 consensus_queue.cc:237] T 550118bc1a0641ce9803bdbf43f2069f P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:55.656841 26754 raft_consensus.cc:729] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c [term 5 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:55.667459 26820 consensus_queue.cc:1035] T b1e85248d3b04b979dd080caeeb02fdb P 1bc1f2e62ab44af995165bd5116a9d43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 4, Time since last communication: 0.000s
I20260925 03:04:55.677392 24437 leader_election.cc:302] T ceab6b0fccd54e909a64f66fe81a9a1f P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:55.695667 24437 leader_election.cc:302] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:04:55.697511 24437 leader_election.cc:302] T c85f5d1d3f004cf5817bb594fd8e6469 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:55.655723 26823 raft_consensus.cc:2726] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20260925 03:04:55.679196 26762 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.665s user 0.002s sys 0.012s
W20260925 03:04:55.698945 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 68810.8us
I20260925 03:04:55.653648 25191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367277daf8d04778b3fa4ed94d71580d" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43"
I20260925 03:04:55.698897 26820 consensus_queue.cc:1225] T b1e85248d3b04b979dd080caeeb02fdb P 1bc1f2e62ab44af995165bd5116a9d43 [LEADER]: Peer eeab93abfdb141a895002ec9f312296c log is divergent from this leader: its last log entry 3.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260925 03:04:55.706764 24437 leader_election.cc:302] T 100c113a9fd84cc8adf2ec7b33ea0093 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:55.638551 26818 consensus_queue.cc:260] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:55.690160 26882 raft_consensus.cc:2726] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260925 03:04:55.681224 26824 consensus_queue.cc:1159] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "eeab93abfdb141a895002ec9f312296c" responder_term: 4 status { last_received { term: 4 index: 3 } last_committed_idx: 2 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 3. Current term is 4. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260925 03:04:55.708472 24621 pending_rounds.cc:76] T b1e85248d3b04b979dd080caeeb02fdb P eeab93abfdb141a895002ec9f312296c: Aborting all ops after (but not including) 4
I20260925 03:04:55.722569 24465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcf6e6a422d4d198d4fbb0f7d3d845d" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:55.722919 24467 raft_consensus.cc:1226] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20260925 03:04:55.723959 24465 raft_consensus.cc:2454] T edcf6e6a422d4d198d4fbb0f7d3d845d P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 4.
I20260925 03:04:55.724483 24468 raft_consensus.cc:1226] T 37a157996e944db2906110d87ecb0c54 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 5. Current term is 6. Ops: []
I20260925 03:04:55.718575 24621 pending_rounds.cc:97] T b1e85248d3b04b979dd080caeeb02fdb P eeab93abfdb141a895002ec9f312296c: Aborting uncommitted NO_OP operation due to leader change: 3.5
I20260925 03:04:55.725849 24621 raft_consensus.cc:2866] T b1e85248d3b04b979dd080caeeb02fdb P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260925 03:04:55.729298 24469 raft_consensus.cc:1226] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 4. Current term is 5. Ops: []
I20260925 03:04:55.730141 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367277daf8d04778b3fa4ed94d71580d" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:55.730588 24540 raft_consensus.cc:3037] T 367277daf8d04778b3fa4ed94d71580d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:04:55.704895 26682 raft_consensus.cc:2726] T ceab6b0fccd54e909a64f66fe81a9a1f P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260925 03:04:55.731576 24473 raft_consensus.cc:1261] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260925 03:04:55.731803 24471 raft_consensus.cc:1226] T 7b8e0d260a884ecdb20f79de46c36b19 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 4. Current term is 5. Ops: []
I20260925 03:04:55.733655 24543 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 1011) took 1379 ms. Trace:
I20260925 03:04:55.733129 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc2df40db08649c9a75e90000b878559" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:55.735246 24548 raft_consensus.cc:3037] T cc2df40db08649c9a75e90000b878559 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Advancing to term 6
I20260925 03:04:55.691087 26754 consensus_queue.cc:260] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:55.736481 26754 raft_consensus.cc:3037] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:04:55.736306 26841 consensus_queue.cc:1035] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.016s
I20260925 03:04:55.740461 26712 raft_consensus.cc:3032] T c88cbd39e01f44628a094af516be2833 P eeab93abfdb141a895002ec9f312296c [term 5 LEADER]: Stepping down as leader of term 5
I20260925 03:04:55.740984 26712 raft_consensus.cc:729] T c88cbd39e01f44628a094af516be2833 P eeab93abfdb141a895002ec9f312296c [term 5 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:55.741860 26712 consensus_queue.cc:260] T c88cbd39e01f44628a094af516be2833 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:55.718303 26378 consensus_queue.cc:237] T 71f461a6e2ff434993b78a2d077f0abd P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:55.749073 26852 raft_consensus.cc:2726] T 100c113a9fd84cc8adf2ec7b33ea0093 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260925 03:04:55.749168 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367277daf8d04778b3fa4ed94d71580d" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:55.734799 24543 rpcz_store.cc:276] 0925 03:04:54.354134 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:54.937220 (+583086us) service_pool.cc:227] Handling call
0925 03:04:54.953856 (+ 16636us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:04:54.954166 (+ 310us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:54.954192 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:04:54.954529 (+ 337us) log.cc:842] Serialized 26 byte log entry
0925 03:04:55.066068 (+111539us) raft_consensus.cc:1631] Marking committed up to 4
0925 03:04:55.066180 (+ 112us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:55.066252 (+ 72us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:04:55.418822 (+352570us) raft_consensus.cc:1671] finished
0925 03:04:55.418846 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:55.723278 (+304432us) spinlock_profiling.cc:243] Waited 16.3 ms on lock 0x7b540010d950. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764cfa09 00007f437b4112e9 00007f4374af7684 00007f4374af7612 00007f4374af7541 00007f4374af74bc 00007f4374af6772 00007f43748fa94c 00007f43748f9d86 00007f43748f972f
0925 03:04:55.733596 (+ 10318us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1039,"raft.run_cpu_time_us":78,"raft.run_wall_time_us":82,"spinlock_wait_cycles":37412736,"thread_start_us":86544,"threads_started":1,"wal-append.queue_time_us":123638}
I20260925 03:04:55.717134 26821 raft_consensus.cc:2726] T c85f5d1d3f004cf5817bb594fd8e6469 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260925 03:04:55.706982 26738 leader_election.cc:288] T cc2df40db08649c9a75e90000b878559 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 6 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:55.764901 24615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b030d84c670b48d5bff337ccc0c77626" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:55.770727 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcf6e6a422d4d198d4fbb0f7d3d845d" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:55.777932 24618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 8 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:55.780509 24472 raft_consensus.cc:1226] T 25102ba08e4e49388f6176badcd10a25 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 3. Current term is 4. Ops: []
I20260925 03:04:55.780786 24618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f68804eb5a244d87bb395d50683c6012" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:55.780993 24615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f22ddd7d1c444dc58f45a1fb0b3e02bc" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:55.781145 24616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06f4a7eaf6534e4986c9fd35911038e4" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:55.781405 24618 raft_consensus.cc:2454] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 5.
I20260925 03:04:55.781690 24614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "550118bc1a0641ce9803bdbf43f2069f" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:55.782370 24614 raft_consensus.cc:2354] T 550118bc1a0641ce9803bdbf43f2069f P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 5. Current term is 6.
I20260925 03:04:55.783474 24622 raft_consensus.cc:1261] T 04e068a0f80941dc9138cee063f53bab P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 6. (term mismatch)
I20260925 03:04:55.783802 24622 pending_rounds.cc:76] T 04e068a0f80941dc9138cee063f53bab P eeab93abfdb141a895002ec9f312296c: Aborting all ops after (but not including) 5
I20260925 03:04:55.783993 24622 pending_rounds.cc:97] T 04e068a0f80941dc9138cee063f53bab P eeab93abfdb141a895002ec9f312296c: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20260925 03:04:55.784204 24622 raft_consensus.cc:2866] T 04e068a0f80941dc9138cee063f53bab P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260925 03:04:55.785527 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f22ddd7d1c444dc58f45a1fb0b3e02bc" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:55.786135 24473 raft_consensus.cc:2454] T f22ddd7d1c444dc58f45a1fb0b3e02bc P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 6.
I20260925 03:04:55.780676 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d89396576884f7c8901dba54f783947" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:55.759089 26841 raft_consensus.cc:979] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c: : Instructing follower a04a37fc4aa7460cbfdbeeb4e451e1f5 to start an election
I20260925 03:04:55.791193 24511 leader_election.cc:302] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 1bc1f2e62ab44af995165bd5116a9d43, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:55.791770 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:55.792209 24511 leader_election.cc:302] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:55.734485 26882 raft_consensus.cc:3032] T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 LEADER]: Stepping down as leader of term 5
I20260925 03:04:55.714087 26741 consensus_queue.cc:1035] T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.050s
I20260925 03:04:55.815230 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7691c6b475428c82d6181d693d26f3" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:55.817293 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b030d84c670b48d5bff337ccc0c77626" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:55.818004 24470 raft_consensus.cc:3037] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Advancing to term 8
I20260925 03:04:55.769018 26818 raft_consensus.cc:3037] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:55.812191 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71f461a6e2ff434993b78a2d077f0abd" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:55.813107 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ac466793d094325ace224708b2158ed" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:55.814165 24511 leader_election.cc:398] T 550118bc1a0641ce9803bdbf43f2069f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 election: Vote denied by peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105) with higher term. Message: Invalid argument: T 550118bc1a0641ce9803bdbf43f2069f P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 5. Current term is 6.
I20260925 03:04:55.822952 24511 leader_election.cc:401] T 550118bc1a0641ce9803bdbf43f2069f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20260925 03:04:55.762929 26712 raft_consensus.cc:3037] T c88cbd39e01f44628a094af516be2833 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Advancing to term 6
W20260925 03:04:55.810809 26891 log.cc:924] Time spent T 37a157996e944db2906110d87ecb0c54 P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.123s user 0.001s sys 0.000s
W20260925 03:04:55.810259 26894 log.cc:924] Time spent T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.057s user 0.000s sys 0.000s
I20260925 03:04:55.798895 24616 raft_consensus.cc:2454] T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 7.
I20260925 03:04:55.827450 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcf6e6a422d4d198d4fbb0f7d3d845d" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:55.833509 24472 raft_consensus.cc:3037] T edcf6e6a422d4d198d4fbb0f7d3d845d P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:55.835211 24511 leader_election.cc:302] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 1bc1f2e62ab44af995165bd5116a9d43, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:55.809932 26738 raft_consensus.cc:482] T 04e068a0f80941dc9138cee063f53bab P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:55.838482 24511 leader_election.cc:302] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:55.814043 26882 raft_consensus.cc:729] T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:04:55.816335 26800 raft_consensus.cc:2781] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260925 03:04:55.816891 26816 raft_consensus.cc:2726] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260925 03:04:55.851678 26897 raft_consensus.cc:2781] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260925 03:04:55.852182 26897 raft_consensus.cc:482] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:55.852545 26897 raft_consensus.cc:3037] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:04:55.846666 26758 raft_consensus.cc:1067] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c [term 5 LEADER]: Signalling peer a04a37fc4aa7460cbfdbeeb4e451e1f5 to start an election
I20260925 03:04:55.856925 24472 raft_consensus.cc:2454] T edcf6e6a422d4d198d4fbb0f7d3d845d P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 5.
I20260925 03:04:55.841218 26719 consensus_queue.cc:1035] T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.019s
I20260925 03:04:55.858646 24471 raft_consensus.cc:1261] T 0be4e0aa984f4f2b81a58e382e434ad5 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260925 03:04:55.843933 26738 raft_consensus.cc:504] T 04e068a0f80941dc9138cee063f53bab P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:55.862723 26738 leader_election.cc:288] T 04e068a0f80941dc9138cee063f53bab P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:55.863823 25191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e068a0f80941dc9138cee063f53bab" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:55.835469 26810 raft_consensus.cc:3037] T 550118bc1a0641ce9803bdbf43f2069f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:04:55.842521 24540 raft_consensus.cc:2454] T 367277daf8d04778b3fa4ed94d71580d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 6.
I20260925 03:04:55.771317 26820 consensus_queue.cc:1046] T 367277daf8d04778b3fa4ed94d71580d P 1bc1f2e62ab44af995165bd5116a9d43 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.058s
I20260925 03:04:55.839232 26807 raft_consensus.cc:2726] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260925 03:04:55.836457 24548 raft_consensus.cc:2454] T cc2df40db08649c9a75e90000b878559 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 6.
I20260925 03:04:55.878759 24470 raft_consensus.cc:2454] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 8.
W20260925 03:04:55.879422 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 172735us
I20260925 03:04:55.845427 26646 raft_consensus.cc:2781] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Leader election won for term 7
I20260925 03:04:55.884191 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e068a0f80941dc9138cee063f53bab" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:55.884953 26646 raft_consensus.cc:686] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 LEADER]: Becoming Leader. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:04:55.885380 26907 consensus_queue.cc:1046] T e6b6ec010caf4689b8c90449764f5cf9 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260925 03:04:55.885917 26646 consensus_queue.cc:237] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:55.886771 26881 raft_consensus.cc:3032] T e6b6ec010caf4689b8c90449764f5cf9 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 LEADER]: Stepping down as leader of term 4
I20260925 03:04:55.887199 26881 raft_consensus.cc:729] T e6b6ec010caf4689b8c90449764f5cf9 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:04:55.888017 26881 consensus_queue.cc:260] T e6b6ec010caf4689b8c90449764f5cf9 P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:55.889705 26881 raft_consensus.cc:3037] T e6b6ec010caf4689b8c90449764f5cf9 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:55.881286 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b8e0d260a884ecdb20f79de46c36b19" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:55.894743 24471 raft_consensus.cc:3037] T 7b8e0d260a884ecdb20f79de46c36b19 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:04:55.895768 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f22ddd7d1c444dc58f45a1fb0b3e02bc" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:55.896399 24470 raft_consensus.cc:3037] T f22ddd7d1c444dc58f45a1fb0b3e02bc P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:04:55.847586 26800 raft_consensus.cc:482] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:55.897724 26897 raft_consensus.cc:504] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:55.910979 26897 leader_election.cc:288] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:55.924952 26882 consensus_queue.cc:260] T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:55.931610 24471 raft_consensus.cc:2454] T 7b8e0d260a884ecdb20f79de46c36b19 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 6.
I20260925 03:04:55.938540 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:55.940418 24467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37a157996e944db2906110d87ecb0c54" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:55.941188 24467 raft_consensus.cc:3037] T 37a157996e944db2906110d87ecb0c54 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:04:55.943676 24613 raft_consensus.cc:2396] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260925 03:04:55.946421 24464 raft_consensus.cc:1261] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260925 03:04:55.942263 24616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ac285175464fbda67211d9574db415" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:55.949558 25191 raft_consensus.cc:1226] T b1e85248d3b04b979dd080caeeb02fdb P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 LEADER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 5. Current term is 6. Ops: []
W20260925 03:04:55.946349 24573 heartbeater.cc:644] Failed to heartbeat to 127.22.193.62:40353 (0 consecutive failures): Timed out: Failed to send heartbeat to master: TSHeartbeat RPC to 127.22.193.62:40353 timed out after 15.000s (SENT)
I20260925 03:04:55.927294 26800 raft_consensus.cc:3037] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:04:55.961123 24620 raft_consensus.cc:1226] T 367277daf8d04778b3fa4ed94d71580d P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Rejecting Update request from peer 1bc1f2e62ab44af995165bd5116a9d43 for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20260925 03:04:55.973207 24616 raft_consensus.cc:1261] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Refusing update from remote peer 1bc1f2e62ab44af995165bd5116a9d43: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260925 03:04:55.938812 26882 raft_consensus.cc:3037] T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:04:55.983358 24464 raft_consensus.cc:1261] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260925 03:04:55.984391 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc2df40db08649c9a75e90000b878559" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:55.984990 24464 raft_consensus.cc:3037] T cc2df40db08649c9a75e90000b878559 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:04:55.986253 24469 raft_consensus.cc:2454] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 7.
I20260925 03:04:55.992930 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367277daf8d04778b3fa4ed94d71580d" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:55.993577 24469 raft_consensus.cc:3037] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:04:55.969821 26846 raft_consensus.cc:504] T 1414dcfe53f64a78ae413531eb8ec117 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:55.996860 26810 raft_consensus.cc:2726] T 550118bc1a0641ce9803bdbf43f2069f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105) with higher term. Message: Invalid argument: T 550118bc1a0641ce9803bdbf43f2069f P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 5. Current term is 6.
I20260925 03:04:55.998181 26846 leader_election.cc:288] T 1414dcfe53f64a78ae413531eb8ec117 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:56.010362 24467 raft_consensus.cc:2454] T 37a157996e944db2906110d87ecb0c54 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 7.
I20260925 03:04:56.011931 24467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1414dcfe53f64a78ae413531eb8ec117" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:56.012681 24467 raft_consensus.cc:3037] T 1414dcfe53f64a78ae413531eb8ec117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Advancing to term 4
W20260925 03:04:56.015920 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 61983.6us
I20260925 03:04:56.022593 24471 raft_consensus.cc:1226] T 8e5819c15eac4544a57e5ee6c9d4f117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260925 03:04:56.021117 24470 raft_consensus.cc:2454] T f22ddd7d1c444dc58f45a1fb0b3e02bc P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 7.
I20260925 03:04:56.008412 24469 raft_consensus.cc:2454] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 6.
W20260925 03:04:56.059134 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 57483us
I20260925 03:04:56.064487 24472 raft_consensus.cc:1261] T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Refusing update from remote peer 1bc1f2e62ab44af995165bd5116a9d43: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260925 03:04:56.067803 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60d45b7d467d4e1ab92367f6d7d52b15" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:55.976624 26800 raft_consensus.cc:504] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:56.073342 24586 leader_election.cc:302] T cc2df40db08649c9a75e90000b878559 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:56.079401 24466 raft_consensus.cc:1226] T 550118bc1a0641ce9803bdbf43f2069f P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 LEADER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 4. Current term is 6. Ops: []
I20260925 03:04:56.079437 24467 raft_consensus.cc:2454] T 1414dcfe53f64a78ae413531eb8ec117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 4.
I20260925 03:04:56.083719 24464 raft_consensus.cc:2454] T cc2df40db08649c9a75e90000b878559 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 6.
I20260925 03:04:56.086225 24514 leader_election.cc:302] T 420163a6cdce412ebc21c25fe3835638 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:56.087126 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367277daf8d04778b3fa4ed94d71580d" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:56.099116 24547 raft_consensus.cc:2354] T 367277daf8d04778b3fa4ed94d71580d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 5. Current term is 6.
I20260925 03:04:56.060580 26803 consensus_queue.cc:1046] T 367277daf8d04778b3fa4ed94d71580d P 1bc1f2e62ab44af995165bd5116a9d43 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: INVALID_TERM, Last received: 4.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.023s
I20260925 03:04:56.104462 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74443a0a1dc34d88a2f42e3b722db405" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:56.105173 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e725b82985746b39c0cc8f671c95353" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:56.103103 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1414dcfe53f64a78ae413531eb8ec117" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:56.106545 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f22ddd7d1c444dc58f45a1fb0b3e02bc" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:56.106372 24588 leader_election.cc:302] T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:04:56.107362 24547 raft_consensus.cc:2354] T f22ddd7d1c444dc58f45a1fb0b3e02bc P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 6. Current term is 7.
I20260925 03:04:56.105255 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7aa47c48404465b94a36fd819fe3c7a" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:56.105898 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9357d144aa49789faaf31fe5e6bf89" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:56.108166 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b8e0d260a884ecdb20f79de46c36b19" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:56.107733 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b030d84c670b48d5bff337ccc0c77626" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:56.107050 24545 raft_consensus.cc:3037] T 1414dcfe53f64a78ae413531eb8ec117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:56.107198 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5332a5d329493991dff994e70ea92c" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:56.110436 24541 raft_consensus.cc:2354] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 5. Current term is 6.
I20260925 03:04:56.109756 24542 raft_consensus.cc:1261] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 4. (term mismatch)
I20260925 03:04:56.111634 24542 pending_rounds.cc:76] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Aborting all ops after (but not including) 3
I20260925 03:04:56.111864 24542 pending_rounds.cc:97] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Aborting uncommitted NO_OP operation due to leader change: 6.4
I20260925 03:04:56.112124 24542 raft_consensus.cc:2866] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260925 03:04:56.102782 25193 raft_consensus.cc:1261] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [term 8 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260925 03:04:56.114784 24544 raft_consensus.cc:2354] T b030d84c670b48d5bff337ccc0c77626 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 7. Current term is 8.
I20260925 03:04:56.074533 26741 consensus_queue.cc:1035] T 420163a6cdce412ebc21c25fe3835638 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.030s
I20260925 03:04:56.117952 25191 raft_consensus.cc:1261] T 29637f5a4a504ce6ba29033ccd0cd716 P 1bc1f2e62ab44af995165bd5116a9d43 [term 7 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260925 03:04:56.119892 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37a157996e944db2906110d87ecb0c54" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:56.120586 24543 raft_consensus.cc:2354] T 37a157996e944db2906110d87ecb0c54 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 6. Current term is 7.
I20260925 03:04:56.126171 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4ebd2b043024a548005c8b6b00b0449" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:56.126937 24539 raft_consensus.cc:2354] T b4ebd2b043024a548005c8b6b00b0449 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 3. Current term is 4.
I20260925 03:04:56.115792 24469 raft_consensus.cc:1261] T edcf6e6a422d4d198d4fbb0f7d3d845d P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260925 03:04:56.128226 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcf6e6a422d4d198d4fbb0f7d3d845d" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:56.104466 26807 raft_consensus.cc:2726] T 420163a6cdce412ebc21c25fe3835638 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260925 03:04:56.128856 24542 raft_consensus.cc:2354] T edcf6e6a422d4d198d4fbb0f7d3d845d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 4. Current term is 5.
I20260925 03:04:56.083609 26577 consensus_queue.cc:1046] T 25102ba08e4e49388f6176badcd10a25 P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.031s
I20260925 03:04:56.105176 26699 raft_consensus.cc:2781] T cc2df40db08649c9a75e90000b878559 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Leader election won for term 6
I20260925 03:04:56.115576 26803 consensus_queue.cc:1035] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.016s
I20260925 03:04:56.133150 24588 leader_election.cc:302] T 367277daf8d04778b3fa4ed94d71580d P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters: 1bc1f2e62ab44af995165bd5116a9d43
I20260925 03:04:56.133836 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "550118bc1a0641ce9803bdbf43f2069f" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:56.115942 24586 leader_election.cc:302] T 04e068a0f80941dc9138cee063f53bab P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 1bc1f2e62ab44af995165bd5116a9d43, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:56.135783 26846 consensus_queue.cc:1035] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260925 03:04:56.138855 26800 leader_election.cc:288] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 6 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:56.129709 26909 raft_consensus.cc:2726] T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260925 03:04:56.113241 26722 consensus_queue.cc:1035] T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.027s
I20260925 03:04:56.141157 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7691c6b475428c82d6181d693d26f3" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:56.131014 26712 consensus_queue.cc:1035] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.014s
I20260925 03:04:56.144994 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7993f9c3e2a44f408bbf5d1e395e2306" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:56.145334 26712 consensus_queue.cc:1225] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer 869b0a8575b54d08ae2817fad5dd42d5 log is divergent from this leader: its last log entry 8.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260925 03:04:56.146132 24548 raft_consensus.cc:1226] T 5278a60f97da4e10aecd50a8d9a9a43c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 3. Current term is 4. Ops: []
I20260925 03:04:56.146729 26508 consensus_queue.cc:1035] T 5278a60f97da4e10aecd50a8d9a9a43c P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.013s
I20260925 03:04:56.148335 24543 raft_consensus.cc:1226] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 6. Current term is 7. Ops: []
I20260925 03:04:56.148777 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:56.149456 24548 raft_consensus.cc:2454] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 7.
I20260925 03:04:56.151301 24544 raft_consensus.cc:2454] T 550118bc1a0641ce9803bdbf43f2069f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 6.
I20260925 03:04:56.154006 24539 raft_consensus.cc:2454] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 7.
I20260925 03:04:56.155155 24545 raft_consensus.cc:2454] T 1414dcfe53f64a78ae413531eb8ec117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 3.
I20260925 03:04:56.162470 24511 leader_election.cc:302] T 8ac466793d094325ace224708b2158ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 1bc1f2e62ab44af995165bd5116a9d43, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:56.160787 26864 raft_consensus.cc:2726] T 04e068a0f80941dc9138cee063f53bab P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260925 03:04:56.164671 26909 raft_consensus.cc:2781] T 367277daf8d04778b3fa4ed94d71580d P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Leader election won for term 6
I20260925 03:04:56.165254 26909 raft_consensus.cc:686] T 367277daf8d04778b3fa4ed94d71580d P eeab93abfdb141a895002ec9f312296c [term 6 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:56.165674 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f68804eb5a244d87bb395d50683c6012" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:56.166420 24613 raft_consensus.cc:3037] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:04:56.170478 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21c43af5e6844fdaacc8c625fa01fd8d" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:56.171797 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f68804eb5a244d87bb395d50683c6012" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:56.172546 24469 raft_consensus.cc:2354] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 4. Current term is 5.
I20260925 03:04:56.169178 26800 raft_consensus.cc:2726] T 8ac466793d094325ace224708b2158ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260925 03:04:56.176970 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ac285175464fbda67211d9574db415" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:56.166594 26699 raft_consensus.cc:686] T cc2df40db08649c9a75e90000b878559 P eeab93abfdb141a895002ec9f312296c [term 6 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
W20260925 03:04:56.178534 26917 log.cc:924] Time spent T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5: Append to log took a long time: real 0.173s user 0.002s sys 0.000s
I20260925 03:04:56.183388 26577 raft_consensus.cc:3032] T 25102ba08e4e49388f6176badcd10a25 P eeab93abfdb141a895002ec9f312296c [term 3 LEADER]: Stepping down as leader of term 3
I20260925 03:04:56.183871 26577 raft_consensus.cc:729] T 25102ba08e4e49388f6176badcd10a25 P eeab93abfdb141a895002ec9f312296c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:56.191532 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74443a0a1dc34d88a2f42e3b722db405" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:56.195305 26699 consensus_queue.cc:237] T cc2df40db08649c9a75e90000b878559 P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:04:56.196442 24514 leader_election.cc:334] T 37a157996e944db2906110d87ecb0c54 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187): Timed out: RequestConsensusVote RPC to 127.22.193.1:42187 timed out after 1.804s (SENT)
I20260925 03:04:56.196930 24468 raft_consensus.cc:1261] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Refusing update from remote peer 1bc1f2e62ab44af995165bd5116a9d43: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
W20260925 03:04:56.197862 26921 log.cc:924] Time spent T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.069s user 0.000s sys 0.000s
I20260925 03:04:56.196933 24514 leader_election.cc:302] T 37a157996e944db2906110d87ecb0c54 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:56.166385 26909 consensus_queue.cc:237] T 367277daf8d04778b3fa4ed94d71580d P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 6, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:56.198973 26800 raft_consensus.cc:2726] T 37a157996e944db2906110d87ecb0c54 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260925 03:04:56.207031 25162 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 183618us
I20260925 03:04:56.208276 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "293ef69caf514ac98510a68495f54eda" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:56.209053 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7aa47c48404465b94a36fd819fe3c7a" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:56.212286 24613 raft_consensus.cc:2454] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 6.
I20260925 03:04:56.215447 24466 raft_consensus.cc:2379] T a7aa47c48404465b94a36fd819fe3c7a P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 5: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:04:56.217574 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcf6e6a422d4d198d4fbb0f7d3d845d" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:56.218530 24469 raft_consensus.cc:2379] T edcf6e6a422d4d198d4fbb0f7d3d845d P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 5: Already voted for candidate eeab93abfdb141a895002ec9f312296c in this term.
I20260925 03:04:56.220387 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b030d84c670b48d5bff337ccc0c77626" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:56.222443 26843 raft_consensus.cc:3037] T 21c43af5e6844fdaacc8c625fa01fd8d P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:56.222648 24539 raft_consensus.cc:1261] T 37a157996e944db2906110d87ecb0c54 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260925 03:04:56.224288 26831 raft_consensus.cc:3032] T 367277daf8d04778b3fa4ed94d71580d P 1bc1f2e62ab44af995165bd5116a9d43 [term 5 LEADER]: Stepping down as leader of term 5
I20260925 03:04:56.224764 26831 raft_consensus.cc:729] T 367277daf8d04778b3fa4ed94d71580d P 1bc1f2e62ab44af995165bd5116a9d43 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1bc1f2e62ab44af995165bd5116a9d43, State: Running, Role: LEADER
I20260925 03:04:56.207084 26577 consensus_queue.cc:260] T 25102ba08e4e49388f6176badcd10a25 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:56.226531 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "550118bc1a0641ce9803bdbf43f2069f" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:56.224291 24545 raft_consensus.cc:1261] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260925 03:04:56.227561 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 8 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:56.228524 24468 raft_consensus.cc:2396] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for term 8 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20260925 03:04:56.230160 24548 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9e8af2641e124f9589c5ccfdac785f63"
dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
from {username='slave'} at 127.0.0.1:36786
I20260925 03:04:56.232972 24548 raft_consensus.cc:482] T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 03:04:56.233309 24548 raft_consensus.cc:3037] T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:04:56.234350 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d89396576884f7c8901dba54f783947" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:56.237339 24465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35992316cb8d4c028333388eff02ecde" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:56.238368 24465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9c33fd100ff44c1a3247319a324a044" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:56.239019 24465 raft_consensus.cc:2379] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 6: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:04:56.234465 24511 leader_election.cc:302] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:56.239737 24465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c23337a7cb74184a0e57b7095cb6a11" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:56.240482 24465 raft_consensus.cc:2379] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 6: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:04:56.241493 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1414dcfe53f64a78ae413531eb8ec117" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:56.242141 24539 raft_consensus.cc:3037] T 1414dcfe53f64a78ae413531eb8ec117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:56.225646 26577 raft_consensus.cc:3037] T 25102ba08e4e49388f6176badcd10a25 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:56.239741 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c88cbd39e01f44628a094af516be2833" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:56.244515 24465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37a157996e944db2906110d87ecb0c54" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:56.245157 24465 raft_consensus.cc:2354] T 37a157996e944db2906110d87ecb0c54 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 6. Current term is 7.
I20260925 03:04:56.245708 24465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be4e0aa984f4f2b81a58e382e434ad5" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:56.246308 24465 raft_consensus.cc:2454] T 0be4e0aa984f4f2b81a58e382e434ad5 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 5.
I20260925 03:04:56.247040 24465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5278a60f97da4e10aecd50a8d9a9a43c" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:56.247570 24465 raft_consensus.cc:2454] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 4.
I20260925 03:04:56.250514 24466 raft_consensus.cc:2379] T c88cbd39e01f44628a094af516be2833 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 6: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:04:56.252467 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7691c6b475428c82d6181d693d26f3" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:56.253118 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6b6ec010caf4689b8c90449764f5cf9" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:56.252355 26828 raft_consensus.cc:482] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:56.255478 26828 raft_consensus.cc:504] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:56.252138 26800 raft_consensus.cc:2781] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader election won for term 6
I20260925 03:04:56.257226 26800 raft_consensus.cc:686] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 LEADER]: Becoming Leader. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:04:56.257692 26828 leader_election.cc:288] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:56.258070 26800 consensus_queue.cc:237] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:56.258764 25194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e92f377a79354a6ebb4eec67303c13e3" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:56.259421 25194 raft_consensus.cc:2454] T e92f377a79354a6ebb4eec67303c13e3 P 1bc1f2e62ab44af995165bd5116a9d43 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 5.
W20260925 03:04:56.226445 26900 consensus_queue.cc:1159] T 25102ba08e4e49388f6176badcd10a25 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" responder_term: 3 status { last_received { term: 3 index: 2 } last_committed_idx: 2 last_received_current_leader { term: 3 index: 2 } }
I20260925 03:04:56.263989 24585 leader_election.cc:302] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:56.270711 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1414dcfe53f64a78ae413531eb8ec117" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:56.271521 24472 raft_consensus.cc:2354] T 1414dcfe53f64a78ae413531eb8ec117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 3. Current term is 4.
I20260925 03:04:56.272343 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5332a5d329493991dff994e70ea92c" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:56.287516 24472 raft_consensus.cc:2379] T 4e5332a5d329493991dff994e70ea92c P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 6: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:04:56.277522 24544 raft_consensus.cc:1226] T 367277daf8d04778b3fa4ed94d71580d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Rejecting Update request from peer 1bc1f2e62ab44af995165bd5116a9d43 for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20260925 03:04:56.278560 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6b6ec010caf4689b8c90449764f5cf9" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:56.279247 26712 raft_consensus.cc:482] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:56.282088 24468 raft_consensus.cc:2454] T e6b6ec010caf4689b8c90449764f5cf9 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 5.
I20260925 03:04:56.289332 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f68804eb5a244d87bb395d50683c6012" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:56.290382 24469 raft_consensus.cc:2362] T e6b6ec010caf4689b8c90449764f5cf9 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 5. Re-sending same reply.
I20260925 03:04:56.290020 24468 raft_consensus.cc:2454] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 5.
I20260925 03:04:56.304224 24539 raft_consensus.cc:2454] T 1414dcfe53f64a78ae413531eb8ec117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 4.
W20260925 03:04:56.304621 24514 leader_election.cc:334] T 1414dcfe53f64a78ae413531eb8ec117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187): Timed out: RequestConsensusVote RPC to 127.22.193.1:42187 timed out after 1.923s (SENT)
I20260925 03:04:56.305161 24514 leader_election.cc:302] T 1414dcfe53f64a78ae413531eb8ec117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:56.305269 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15861cd28e354598a3c6bfc4337c34bb" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:56.305109 24465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4ebd2b043024a548005c8b6b00b0449" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:56.306143 24465 raft_consensus.cc:2379] T b4ebd2b043024a548005c8b6b00b0449 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 4: Already voted for candidate 1bc1f2e62ab44af995165bd5116a9d43 in this term.
I20260925 03:04:56.305333 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43366a0a2f9d4344bf4e37ffd290ef8a" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:56.305948 24469 raft_consensus.cc:2454] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 5.
I20260925 03:04:56.308326 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9357d144aa49789faaf31fe5e6bf89" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:56.331064 24543 raft_consensus.cc:1261] T b030d84c670b48d5bff337ccc0c77626 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20260925 03:04:56.317493 24465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcf6e6a422d4d198d4fbb0f7d3d845d" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:56.336071 24465 raft_consensus.cc:2379] T edcf6e6a422d4d198d4fbb0f7d3d845d P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 5: Already voted for candidate eeab93abfdb141a895002ec9f312296c in this term.
I20260925 03:04:56.309234 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e5819c15eac4544a57e5ee6c9d4f117" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:56.337301 24468 raft_consensus.cc:2379] T 8e5819c15eac4544a57e5ee6c9d4f117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 6: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:04:56.308638 24548 raft_consensus.cc:504] T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:56.336959 24465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "100c113a9fd84cc8adf2ec7b33ea0093" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:56.342057 24545 raft_consensus.cc:1226] T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 5. Current term is 6. Ops: []
I20260925 03:04:56.343407 24541 raft_consensus.cc:1226] T 3d89396576884f7c8901dba54f783947 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 3. Current term is 4. Ops: []
I20260925 03:04:56.345646 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b030d84c670b48d5bff337ccc0c77626" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:56.344595 24548 leader_election.cc:288] T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 6 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:56.349363 26810 raft_consensus.cc:482] T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:56.349898 26810 raft_consensus.cc:504] T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:56.350652 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "550118bc1a0641ce9803bdbf43f2069f" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:56.351728 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5f9b503b0ab43999d582015a367441e" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:56.352396 24470 raft_consensus.cc:2379] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 7: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
W20260925 03:04:56.354012 24436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.4:43979, user_credentials={real_user=slave}} blocked reactor thread for 215661us
I20260925 03:04:56.353902 24542 raft_consensus.cc:1261] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260925 03:04:56.358119 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e92f377a79354a6ebb4eec67303c13e3" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:56.358603 24547 raft_consensus.cc:1261] T f22ddd7d1c444dc58f45a1fb0b3e02bc P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260925 03:04:56.357488 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c85f5d1d3f004cf5817bb594fd8e6469" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:56.354283 26508 consensus_queue.cc:1035] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.008s
I20260925 03:04:56.225703 26831 consensus_queue.cc:260] T 367277daf8d04778b3fa4ed94d71580d P 1bc1f2e62ab44af995165bd5116a9d43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:56.363286 26754 consensus_queue.cc:1035] T 37a157996e944db2906110d87ecb0c54 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260925 03:04:56.364480 24588 leader_election.cc:302] T 1414dcfe53f64a78ae413531eb8ec117 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:56.310204 26829 raft_consensus.cc:2781] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260925 03:04:56.425369 24465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48ca6ccd98194cf2af7b5278d15bf4ea" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:56.358768 24541 raft_consensus.cc:2454] T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 4.
I20260925 03:04:56.349159 26883 raft_consensus.cc:482] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:56.306524 26803 raft_consensus.cc:482] T 8ac466793d094325ace224708b2158ed P 1bc1f2e62ab44af995165bd5116a9d43 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:56.428566 26829 raft_consensus.cc:482] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:56.429418 24465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4ebd2b043024a548005c8b6b00b0449" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:56.334252 26807 raft_consensus.cc:2726] T 1414dcfe53f64a78ae413531eb8ec117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260925 03:04:56.309248 26864 raft_consensus.cc:482] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:56.439055 24465 raft_consensus.cc:2379] T b4ebd2b043024a548005c8b6b00b0449 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 4: Already voted for candidate 1bc1f2e62ab44af995165bd5116a9d43 in this term.
I20260925 03:04:56.439906 24465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442d4a29b1d743599f4b6b4871e26668" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:56.440536 24465 raft_consensus.cc:2454] T 442d4a29b1d743599f4b6b4871e26668 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 6.
I20260925 03:04:56.430675 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b8e0d260a884ecdb20f79de46c36b19" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:56.431628 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b5df303cd6a4d59aca5c18bbbff24ed" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:56.441306 24465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06f4a7eaf6534e4986c9fd35911038e4" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
W20260925 03:04:56.430028 24514 leader_election.cc:334] T 8e5819c15eac4544a57e5ee6c9d4f117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187): Timed out: RequestConsensusVote RPC to 127.22.193.1:42187 timed out after 2.019s (SENT)
I20260925 03:04:56.437484 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e5819c15eac4544a57e5ee6c9d4f117" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:56.442610 24466 raft_consensus.cc:2354] T 8e5819c15eac4544a57e5ee6c9d4f117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 4. Current term is 6.
W20260925 03:04:56.437304 25165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 80127.5us
W20260925 03:04:56.437260 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 71494.5us
I20260925 03:04:56.433666 24621 raft_consensus.cc:1261] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260925 03:04:56.444901 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bb46bce3564841b7674e6a693bd2e6" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:56.445253 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f22ddd7d1c444dc58f45a1fb0b3e02bc" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:56.446003 24469 raft_consensus.cc:2379] T f22ddd7d1c444dc58f45a1fb0b3e02bc P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 7: Already voted for candidate eeab93abfdb141a895002ec9f312296c in this term.
I20260925 03:04:56.446462 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e8af2641e124f9589c5ccfdac785f63" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:56.446715 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e725b82985746b39c0cc8f671c95353" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:56.447377 24469 raft_consensus.cc:2379] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 6: Already voted for candidate eeab93abfdb141a895002ec9f312296c in this term.
I20260925 03:04:56.447549 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7993f9c3e2a44f408bbf5d1e395e2306" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:56.448096 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ac285175464fbda67211d9574db415" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:56.365512 26682 raft_consensus.cc:482] T 06f4a7eaf6534e4986c9fd35911038e4 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:56.426252 26820 consensus_queue.cc:1046] T 367277daf8d04778b3fa4ed94d71580d P 1bc1f2e62ab44af995165bd5116a9d43 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true }, Status: INVALID_TERM, Last received: 4.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.024s
I20260925 03:04:56.449501 24468 raft_consensus.cc:2354] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 5. Current term is 7.
I20260925 03:04:56.448561 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceab6b0fccd54e909a64f66fe81a9a1f" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:56.452262 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "293ef69caf514ac98510a68495f54eda" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:56.454347 24473 raft_consensus.cc:2354] T 7b8e0d260a884ecdb20f79de46c36b19 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 5. Current term is 6.
I20260925 03:04:56.454910 24469 raft_consensus.cc:2379] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 6: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:04:56.293393 26712 raft_consensus.cc:504] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:56.365571 26508 raft_consensus.cc:2781] T 1414dcfe53f64a78ae413531eb8ec117 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader election won for term 4
I20260925 03:04:56.351742 26810 leader_election.cc:288] T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:56.364984 25194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e92f377a79354a6ebb4eec67303c13e3" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:56.365331 26878 consensus_queue.cc:1035] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.125s
I20260925 03:04:56.429755 26803 raft_consensus.cc:504] T 8ac466793d094325ace224708b2158ed P 1bc1f2e62ab44af995165bd5116a9d43 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:56.456233 26861 raft_consensus.cc:482] T b627ecb4a5384ef392aa695e06599c39 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:56.334249 23300 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260925 03:04:56.601254 24621 raft_consensus.cc:1261] T 550118bc1a0641ce9803bdbf43f2069f P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
W20260925 03:04:56.882954 24368 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39374 (request call id 205) took 15944 ms (client timeout 15000 ms). Trace:
I20260925 03:04:56.509950 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7aa47c48404465b94a36fd819fe3c7a" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:56.429925 26829 raft_consensus.cc:3037] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:04:56.492463 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06f4a7eaf6534e4986c9fd35911038e4" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:56.695616 26843 raft_consensus.cc:482] T c88cbd39e01f44628a094af516be2833 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260925 03:04:56.531008 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 87125us
W20260925 03:04:56.779039 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 321322us
W20260925 03:04:56.625113 25162 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 257301us
I20260925 03:04:56.595491 24467 raft_consensus.cc:1261] T 21c43af5e6844fdaacc8c625fa01fd8d P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260925 03:04:56.455308 26858 log.cc:924] Time spent T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.106s user 0.000s sys 0.005s
I20260925 03:04:56.936476 24470 raft_consensus.cc:2379] T a7aa47c48404465b94a36fd819fe3c7a P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 5: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
W20260925 03:04:56.940662 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 510665us
W20260925 03:04:56.952194 24514 leader_election.cc:334] T 100c113a9fd84cc8adf2ec7b33ea0093 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187): Timed out: RequestConsensusVote RPC to 127.22.193.1:42187 timed out after 2.027s (SENT)
I20260925 03:04:56.459794 26807 raft_consensus.cc:482] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:56.953800 24469 raft_consensus.cc:1261] T 7b8e0d260a884ecdb20f79de46c36b19 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260925 03:04:56.929122 25194 raft_consensus.cc:2379] T e92f377a79354a6ebb4eec67303c13e3 P 1bc1f2e62ab44af995165bd5116a9d43 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 5: Already voted for candidate 1bc1f2e62ab44af995165bd5116a9d43 in this term.
I20260925 03:04:56.869168 24371 catalog_manager.cc:5495] T 8ac466793d094325ace224708b2158ed P 1bc1f2e62ab44af995165bd5116a9d43 reported cstate change: term changed from 4 to 5, leader changed from a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2) to eeab93abfdb141a895002ec9f312296c (127.22.193.3). New cstate: current_term: 5 leader_uuid: "eeab93abfdb141a895002ec9f312296c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:56.750198 26696 raft_consensus.cc:482] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:56.587390 26508 raft_consensus.cc:686] T 1414dcfe53f64a78ae413531eb8ec117 P eeab93abfdb141a895002ec9f312296c [term 4 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:56.840305 26800 raft_consensus.cc:482] T 21c43af5e6844fdaacc8c625fa01fd8d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:56.948925 26824 raft_consensus.cc:482] T 21c43af5e6844fdaacc8c625fa01fd8d P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:56.949359 26897 consensus_queue.cc:1035] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.007s
I20260925 03:04:56.961755 26824 raft_consensus.cc:504] T 21c43af5e6844fdaacc8c625fa01fd8d P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:56.545324 26927 consensus_queue.cc:1035] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.508s
I20260925 03:04:56.943755 26874 consensus_queue.cc:1035] T 550118bc1a0641ce9803bdbf43f2069f P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.021s
I20260925 03:04:56.964820 26659 consensus_queue.cc:1035] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260925 03:04:56.966882 26508 consensus_queue.cc:237] T 1414dcfe53f64a78ae413531eb8ec117 P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:56.592092 26712 leader_election.cc:288] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:56.969467 26696 raft_consensus.cc:504] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:56.971592 26803 leader_election.cc:288] T 8ac466793d094325ace224708b2158ed P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:56.961719 26800 raft_consensus.cc:504] T 21c43af5e6844fdaacc8c625fa01fd8d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:56.974972 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ac466793d094325ace224708b2158ed" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:56.975607 24613 raft_consensus.cc:2454] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 6.
I20260925 03:04:56.957696 24470 raft_consensus.cc:1226] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 5. Current term is 6. Ops: []
I20260925 03:04:56.842118 26831 raft_consensus.cc:3037] T 367277daf8d04778b3fa4ed94d71580d P 1bc1f2e62ab44af995165bd5116a9d43 [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:04:56.489491 26741 raft_consensus.cc:482] T 0b5df303cd6a4d59aca5c18bbbff24ed P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:56.468309 26882 raft_consensus.cc:482] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:56.781334 26828 raft_consensus.cc:482] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:56.752645 26509 raft_consensus.cc:482] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:56.794042 26799 raft_consensus.cc:482] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:56.474148 26900 raft_consensus.cc:482] T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:56.489463 26552 raft_consensus.cc:482] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:56.955499 24514 leader_election.cc:302] T 100c113a9fd84cc8adf2ec7b33ea0093 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:56.842690 26804 raft_consensus.cc:482] T a4abe0ea757e4886be4945c5001e529a P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:56.976226 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f1fac028a144058b92331c0d9b44ebf" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:56.429716 26883 raft_consensus.cc:504] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:56.977380 26380 consensus_queue.cc:1035] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260925 03:04:56.781634 26820 raft_consensus.cc:482] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:56.457511 26864 raft_consensus.cc:504] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:04:56.930735 24368 rpcz_store.cc:269] 0925 03:04:40.938917 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:40.939073 (+ 156us) service_pool.cc:227] Handling call
0925 03:04:56.882842 (+15943769us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0925 03:04:54.837055 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0925 03:04:54.837670 (+ 615us) write_op.cc:432] Acquiring schema lock in shared mode
0925 03:04:54.837720 (+ 50us) write_op.cc:435] Acquired schema lock
0925 03:04:54.837732 (+ 12us) tablet.cc:660] Decoding operations
0925 03:04:54.838744 (+ 1012us) write_op.cc:620] Acquiring the partition lock for write op
0925 03:04:54.838829 (+ 85us) write_op.cc:641] Partition lock acquired for write op
0925 03:04:54.838855 (+ 26us) tablet.cc:683] Acquiring locks for 10 operations
0925 03:04:54.839714 (+ 859us) tablet.cc:699] Row locks acquired
0925 03:04:54.839749 (+ 35us) write_op.cc:260] PREPARE: finished
0925 03:04:54.839913 (+ 164us) write_op.cc:270] Start()
0925 03:04:54.840057 (+ 144us) write_op.cc:276] Timestamp: P: 1790305494839854 usec, L: 0
0925 03:04:54.840090 (+ 33us) op_driver.cc:331] REPLICATION: starting
0925 03:04:54.941349 (+101259us) log.cc:842] Serialized 3993 byte log entry
0925 03:04:55.334610 (+393261us) op_driver.cc:444] REPLICATION: finished
0925 03:04:55.497469 (+162859us) write_op.cc:301] APPLY: starting
0925 03:04:55.497569 (+ 100us) tablet.cc:1365] starting BulkCheckPresence
0925 03:04:55.497786 (+ 217us) tablet.cc:1368] finished BulkCheckPresence
0925 03:04:55.497909 (+ 123us) tablet.cc:1370] starting ApplyRowOperation cycle
0925 03:04:55.499167 (+ 1258us) tablet.cc:1381] finished ApplyRowOperation cycle
0925 03:04:55.499447 (+ 280us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=10
0925 03:04:55.499486 (+ 39us) write_op.cc:312] APPLY: finished
0925 03:04:55.499828 (+ 342us) log.cc:842] Serialized 76 byte log entry
0925 03:04:55.500000 (+ 172us) write_op.cc:489] Releasing partition, row and schema locks
0925 03:04:55.500122 (+ 122us) write_op.cc:454] Released schema lock
0925 03:04:55.500181 (+ 59us) write_op.cc:341] FINISH: Updating metrics
0925 03:04:55.513103 (+ 12922us) write_op.cc:489] Releasing partition, row and schema locks
0925 03:04:55.517458 (+ 4355us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":3396,"reported_tablets":35,"child_traces":[["op",{"apply.queue_time_us":162526,"apply.run_cpu_time_us":3546,"apply.run_wall_time_us":26117,"num_ops":10,"prepare.queue_time_us":182887,"prepare.run_cpu_time_us":12797,"prepare.run_wall_time_us":223515,"raft.queue_time_us":187347,"raft.run_cpu_time_us":37624,"raft.run_wall_time_us":119269,"replication_time_us":494415,"thread_start_us":458860,"threads_started":4,"wal-append.queue_time_us":203200,"wal-append.run_cpu_time_us":22637,"wal-append.run_wall_time_us":1400822}]]}
I20260925 03:04:57.043053 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e8af2641e124f9589c5ccfdac785f63" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:56.792392 26922 consensus_queue.cc:1035] T b1e85248d3b04b979dd080caeeb02fdb P 1bc1f2e62ab44af995165bd5116a9d43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.184s
I20260925 03:04:56.458884 26682 raft_consensus.cc:504] T 06f4a7eaf6534e4986c9fd35911038e4 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:57.064229 26682 leader_election.cc:288] T 06f4a7eaf6534e4986c9fd35911038e4 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:57.065528 24465 raft_consensus.cc:2396] T 06f4a7eaf6534e4986c9fd35911038e4 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20260925 03:04:56.991936 25162 leader_election.cc:302] T 8ac466793d094325ace224708b2158ed P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:56.978076 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "550118bc1a0641ce9803bdbf43f2069f" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:56.960932 24371 catalog_manager.cc:5495] T 367277daf8d04778b3fa4ed94d71580d P 1bc1f2e62ab44af995165bd5116a9d43 reported cstate change: config changed from index 2 to 5, term changed from 4 to 5, leader changed from eeab93abfdb141a895002ec9f312296c (127.22.193.3) to 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) changed from NON_VOTER to VOTER. New cstate: current_term: 5 leader_uuid: "1bc1f2e62ab44af995165bd5116a9d43" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260925 03:04:57.053454 24466 raft_consensus.cc:2379] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 6: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:04:56.570289 26758 consensus_queue.cc:1035] T edcf6e6a422d4d198d4fbb0f7d3d845d P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.409s
I20260925 03:04:57.057695 24613 raft_consensus.cc:3032] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c [term 5 LEADER]: Stepping down as leader of term 5
I20260925 03:04:57.058689 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e92f377a79354a6ebb4eec67303c13e3" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:57.055029 26509 raft_consensus.cc:504] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:57.058743 26749 consensus_queue.cc:1046] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260925 03:04:57.055305 26741 raft_consensus.cc:504] T 0b5df303cd6a4d59aca5c18bbbff24ed P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:57.059468 26810 raft_consensus.cc:2726] T 100c113a9fd84cc8adf2ec7b33ea0093 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260925 03:04:57.060389 26722 raft_consensus.cc:482] T a7aa47c48404465b94a36fd819fe3c7a P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:57.076673 24371 catalog_manager.cc:5495] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43 reported cstate change: term changed from 3 to 4, leader changed from a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2) to 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4). New cstate: current_term: 4 leader_uuid: "1bc1f2e62ab44af995165bd5116a9d43" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260925 03:04:56.954821 26843 raft_consensus.cc:504] T c88cbd39e01f44628a094af516be2833 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:57.079916 24464 pending_rounds.cc:76] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5: Aborting all ops after (but not including) 5
I20260925 03:04:57.087498 24464 pending_rounds.cc:97] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5: Aborting uncommitted NO_OP operation due to leader change: 8.6
I20260925 03:04:57.054466 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c23337a7cb74184a0e57b7095cb6a11" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:57.071512 24613 raft_consensus.cc:729] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c [term 5 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:57.087771 24464 raft_consensus.cc:2866] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260925 03:04:57.084255 24371 catalog_manager.cc:5495] T 7e49248643df4131a1c42e74244e05c2 P 1bc1f2e62ab44af995165bd5116a9d43 reported cstate change: term changed from 3 to 6. New cstate: current_term: 6 leader_uuid: "869b0a8575b54d08ae2817fad5dd42d5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:04:57.088461 24469 raft_consensus.cc:2454] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 6.
I20260925 03:04:57.097967 24588 leader_election.cc:302] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
W20260925 03:04:57.083899 25162 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 92078.5us
I20260925 03:04:57.034101 26874 raft_consensus.cc:482] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:57.085839 24616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06f4a7eaf6534e4986c9fd35911038e4" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:57.088930 24613 consensus_queue.cc:260] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:57.101933 24613 raft_consensus.cc:3037] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:04:57.102833 24616 raft_consensus.cc:2454] T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 7.
I20260925 03:04:57.104084 24436 leader_election.cc:302] T 06f4a7eaf6534e4986c9fd35911038e4 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:57.105198 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60d45b7d467d4e1ab92367f6d7d52b15" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:57.105839 24470 raft_consensus.cc:2454] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 7.
I20260925 03:04:57.107040 24588 leader_election.cc:302] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
W20260925 03:04:57.110276 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 158606us
W20260925 03:04:57.110948 24514 leader_election.cc:334] T 0b5df303cd6a4d59aca5c18bbbff24ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187): Timed out: RequestConsensusVote RPC to 127.22.193.1:42187 timed out after 2.035s (SENT)
I20260925 03:04:57.111280 24514 leader_election.cc:302] T 0b5df303cd6a4d59aca5c18bbbff24ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:57.055254 26804 raft_consensus.cc:504] T a4abe0ea757e4886be4945c5001e529a P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:57.019421 26762 consensus_queue.cc:1046] T 3d89396576884f7c8901dba54f783947 P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.079s
I20260925 03:04:57.105620 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f68804eb5a244d87bb395d50683c6012" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:57.124832 24466 raft_consensus.cc:3037] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:04:57.083817 26682 raft_consensus.cc:482] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:57.035544 26378 consensus_queue.cc:1035] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.062s
I20260925 03:04:57.034276 26712 consensus_queue.cc:1035] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.064s
I20260925 03:04:57.113618 26810 raft_consensus.cc:2726] T 0b5df303cd6a4d59aca5c18bbbff24ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260925 03:04:57.123245 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e8af2641e124f9589c5ccfdac785f63" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:57.137020 24469 raft_consensus.cc:3037] T 9e8af2641e124f9589c5ccfdac785f63 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:04:57.118867 24613 raft_consensus.cc:2454] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 6.
I20260925 03:04:56.971701 26696 leader_election.cc:288] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:57.114749 26738 raft_consensus.cc:3032] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [term 5 LEADER]: Stepping down as leader of term 5
W20260925 03:04:57.128373 24514 leader_election.cc:334] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187): Timed out: RequestConsensusVote RPC to 127.22.193.1:42187 timed out after 2.085s (SENT)
I20260925 03:04:57.068274 26820 raft_consensus.cc:504] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:57.129833 24470 raft_consensus.cc:1261] T b4ebd2b043024a548005c8b6b00b0449 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Refusing update from remote peer 1bc1f2e62ab44af995165bd5116a9d43: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260925 03:04:57.132014 24511 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.587s user 0.044s sys 0.045s
I20260925 03:04:57.097373 26722 raft_consensus.cc:504] T a7aa47c48404465b94a36fd819fe3c7a P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:57.056011 26882 raft_consensus.cc:504] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:57.055883 26828 raft_consensus.cc:504] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:57.066169 26552 raft_consensus.cc:504] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:57.053686 26900 raft_consensus.cc:504] T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:57.119839 26754 raft_consensus.cc:2781] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260925 03:04:57.055151 26799 raft_consensus.cc:504] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:57.142014 24514 leader_election.cc:302] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:57.096599 26803 raft_consensus.cc:2781] T 8ac466793d094325ace224708b2158ed P 1bc1f2e62ab44af995165bd5116a9d43 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260925 03:04:57.145828 24511 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.601s user 0.044s sys 0.045s
I20260925 03:04:56.972779 26807 raft_consensus.cc:504] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:04:57.168150 25165 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.699s user 0.012s sys 0.031s
W20260925 03:04:57.177207 25165 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.708s user 0.013s sys 0.031s
W20260925 03:04:57.177511 25165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 733933us
I20260925 03:04:56.955256 26861 raft_consensus.cc:504] T b627ecb4a5384ef392aa695e06599c39 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:57.129560 26293 raft_consensus.cc:2781] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260925 03:04:57.153990 24466 raft_consensus.cc:2454] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 6.
I20260925 03:04:57.187318 24466 raft_consensus.cc:1261] T f22ddd7d1c444dc58f45a1fb0b3e02bc P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260925 03:04:57.141837 26738 raft_consensus.cc:729] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [term 5 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:57.133229 26682 raft_consensus.cc:504] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:57.114111 26874 raft_consensus.cc:504] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:57.182518 26882 leader_election.cc:288] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
W20260925 03:04:57.183804 26931 log.cc:924] Time spent T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.079s user 0.000s sys 0.000s
I20260925 03:04:57.184036 26741 leader_election.cc:288] T 0b5df303cd6a4d59aca5c18bbbff24ed P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:57.184614 26864 leader_election.cc:288] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:57.176080 26913 raft_consensus.cc:2781] T 06f4a7eaf6534e4986c9fd35911038e4 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260925 03:04:57.167413 26754 raft_consensus.cc:482] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:57.176363 24469 raft_consensus.cc:2454] T 9e8af2641e124f9589c5ccfdac785f63 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 6.
W20260925 03:04:57.176808 25164 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.665s user 0.007s sys 0.016s
I20260925 03:04:57.168614 26803 raft_consensus.cc:482] T 8ac466793d094325ace224708b2158ed P 1bc1f2e62ab44af995165bd5116a9d43 [term 5 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:57.245613 24613 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3d89396576884f7c8901dba54f783947"
dest_uuid: "eeab93abfdb141a895002ec9f312296c"
mode: GRACEFUL
new_leader_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
from {username='slave'} at 127.0.0.1:44570
W20260925 03:04:57.187318 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 58987.8us
I20260925 03:04:57.182335 26843 leader_election.cc:288] T c88cbd39e01f44628a094af516be2833 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:57.182456 25194 raft_consensus.cc:1261] T e6b6ec010caf4689b8c90449764f5cf9 P 1bc1f2e62ab44af995165bd5116a9d43 [term 5 FOLLOWER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 5 index: 8. (index mismatch)
I20260925 03:04:57.186280 26810 raft_consensus.cc:2726] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260925 03:04:57.226579 26509 leader_election.cc:288] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:57.247126 24613 raft_consensus.cc:595] T 3d89396576884f7c8901dba54f783947 P eeab93abfdb141a895002ec9f312296c [term 3 LEADER]: Received request to transfer leadership to TS a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:57.223371 26293 raft_consensus.cc:482] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:57.265400 24466 raft_consensus.cc:1261] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260925 03:04:57.239270 26913 raft_consensus.cc:482] T 06f4a7eaf6534e4986c9fd35911038e4 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:57.190778 24511 leader_election.cc:302] T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:57.226501 26758 raft_consensus.cc:482] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:57.232357 26762 raft_consensus.cc:482] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:57.245069 26803 raft_consensus.cc:3037] T 8ac466793d094325ace224708b2158ed P 1bc1f2e62ab44af995165bd5116a9d43 [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:04:57.277011 25196 raft_consensus.cc:1261] T 71f461a6e2ff434993b78a2d077f0abd P 1bc1f2e62ab44af995165bd5116a9d43 [term 7 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260925 03:04:57.272509 24470 raft_consensus.cc:1261] T 37a157996e944db2906110d87ecb0c54 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
W20260925 03:04:57.247064 25164 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.779s user 0.007s sys 0.017s
I20260925 03:04:57.240239 26882 raft_consensus.cc:482] T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:57.243666 26754 raft_consensus.cc:3037] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:04:57.240839 26936 raft_consensus.cc:482] T 7e49248643df4131a1c42e74244e05c2 P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
W20260925 03:04:57.290050 24511 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 99420.6us
W20260925 03:04:57.291883 24513 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.694s user 0.007s sys 0.010s
I20260925 03:04:57.267206 26883 leader_election.cc:288] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
W20260925 03:04:57.298084 24513 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.858s user 0.008s sys 0.011s
I20260925 03:04:57.281770 26293 raft_consensus.cc:3037] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Advancing to term 8
I20260925 03:04:57.271231 26938 raft_consensus.cc:482] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:57.302561 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c23337a7cb74184a0e57b7095cb6a11" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:57.318532 24541 raft_consensus.cc:2454] T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 5.
I20260925 03:04:57.279870 26800 leader_election.cc:288] T 21c43af5e6844fdaacc8c625fa01fd8d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:57.283859 26913 raft_consensus.cc:3037] T 06f4a7eaf6534e4986c9fd35911038e4 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Advancing to term 8
I20260925 03:04:57.305371 24546 raft_consensus.cc:1261] T cc2df40db08649c9a75e90000b878559 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260925 03:04:57.307135 24543 raft_consensus.cc:1261] T edcf6e6a422d4d198d4fbb0f7d3d845d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260925 03:04:57.311156 24470 raft_consensus.cc:1226] T 9e8af2641e124f9589c5ccfdac785f63 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 5. Current term is 6. Ops: []
I20260925 03:04:57.296550 26646 raft_consensus.cc:2781] T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader election won for term 6
I20260925 03:04:57.327643 24540 raft_consensus.cc:1261] T 367277daf8d04778b3fa4ed94d71580d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260925 03:04:57.328277 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4abe0ea757e4886be4945c5001e529a" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:57.333532 24546 raft_consensus.cc:1261] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
W20260925 03:04:57.316968 24436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 212960us
I20260925 03:04:57.341393 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c88cbd39e01f44628a094af516be2833" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:57.342087 24613 raft_consensus.cc:2454] T c88cbd39e01f44628a094af516be2833 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 6.
I20260925 03:04:57.345779 24613 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a4abe0ea757e4886be4945c5001e529a"
dest_uuid: "eeab93abfdb141a895002ec9f312296c"
mode: GRACEFUL
new_leader_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
from {username='slave'} at 127.0.0.1:44570
I20260925 03:04:57.346459 24613 raft_consensus.cc:595] T a4abe0ea757e4886be4945c5001e529a P eeab93abfdb141a895002ec9f312296c [term 2 LEADER]: Received request to transfer leadership to TS a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:57.297914 26509 consensus_queue.cc:1035] T 71f461a6e2ff434993b78a2d077f0abd P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.049s
I20260925 03:04:57.349201 26722 leader_election.cc:288] T a7aa47c48404465b94a36fd819fe3c7a P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:57.353529 26828 leader_election.cc:288] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:57.354866 25196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e54ca1757ee4e42956a9cebbeca4f62" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:57.355522 25196 raft_consensus.cc:2454] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 8.
I20260925 03:04:57.356906 24585 leader_election.cc:302] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:57.291887 26738 consensus_queue.cc:260] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:57.359400 24473 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1185) took 1016 ms. Trace:
I20260925 03:04:57.359810 25193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5aec38e66a947c681e6c2a72b05e6ca" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:57.308421 26936 raft_consensus.cc:504] T 7e49248643df4131a1c42e74244e05c2 P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:57.360911 26820 leader_election.cc:288] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:57.321404 26804 leader_election.cc:288] T a4abe0ea757e4886be4945c5001e529a P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:57.362550 26936 leader_election.cc:288] T 7e49248643df4131a1c42e74244e05c2 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:57.339221 26800 consensus_queue.cc:1035] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.024s
I20260925 03:04:57.363431 24436 leader_election.cc:302] T c88cbd39e01f44628a094af516be2833 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:57.364248 26874 leader_election.cc:288] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:57.365007 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60d45b7d467d4e1ab92367f6d7d52b15" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:57.306425 26882 raft_consensus.cc:504] T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:57.359807 24473 rpcz_store.cc:276] 0925 03:04:56.343442 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:56.473237 (+129795us) service_pool.cc:227] Handling call
0925 03:04:56.473645 (+ 408us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:04:56.491001 (+ 17356us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:56.491036 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:04:56.808564 (+317528us) log.cc:842] Serialized 26 byte log entry
0925 03:04:57.130376 (+321812us) raft_consensus.cc:1631] Marking committed up to 4
0925 03:04:57.157557 (+ 27181us) log.cc:842] Serialized 14 byte log entry
0925 03:04:57.157760 (+ 203us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:57.157834 (+ 74us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:04:57.359204 (+201370us) raft_consensus.cc:1671] finished
0925 03:04:57.359227 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:57.359344 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":16892,"prepare.queue_time_us":28093,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":44,"raft.queue_time_us":147297,"raft.run_cpu_time_us":303,"raft.run_wall_time_us":39481,"thread_start_us":285325,"threads_started":1,"wal-append.queue_time_us":444152}
I20260925 03:04:57.368844 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "293ef69caf514ac98510a68495f54eda" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:57.366703 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06f4a7eaf6534e4986c9fd35911038e4" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:57.369998 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f1fac028a144058b92331c0d9b44ebf" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:57.370600 24473 raft_consensus.cc:2454] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 6.
I20260925 03:04:57.371376 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e54ca1757ee4e42956a9cebbeca4f62" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:57.339823 26938 raft_consensus.cc:504] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:57.384292 26882 leader_election.cc:288] T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:57.384781 26646 raft_consensus.cc:686] T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 LEADER]: Becoming Leader. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:04:57.385352 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7aa47c48404465b94a36fd819fe3c7a" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:57.386004 24473 raft_consensus.cc:2454] T a7aa47c48404465b94a36fd819fe3c7a P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 5.
I20260925 03:04:57.385784 26646 consensus_queue.cc:237] T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:57.386823 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b5df303cd6a4d59aca5c18bbbff24ed" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:57.387424 24473 raft_consensus.cc:2454] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 7.
I20260925 03:04:57.388249 26552 leader_election.cc:288] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:57.388713 26874 raft_consensus.cc:482] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:57.389303 26874 raft_consensus.cc:504] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:57.391404 26874 leader_election.cc:288] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:57.383165 26940 consensus_queue.cc:1035] T b4ebd2b043024a548005c8b6b00b0449 P 1bc1f2e62ab44af995165bd5116a9d43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260925 03:04:57.393421 26380 consensus_queue.cc:1035] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260925 03:04:57.395385 24465 raft_consensus.cc:1261] T e6b6ec010caf4689b8c90449764f5cf9 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 5 index: 8. (index mismatch)
I20260925 03:04:57.395800 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5332a5d329493991dff994e70ea92c" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:57.397508 24543 raft_consensus.cc:2454] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 6.
I20260925 03:04:57.398295 24472 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1183) took 1055 ms. Trace:
I20260925 03:04:57.289629 26758 raft_consensus.cc:504] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:57.293213 26762 raft_consensus.cc:504] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:57.409178 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b5df303cd6a4d59aca5c18bbbff24ed" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
W20260925 03:04:57.412186 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 71062.5us
I20260925 03:04:57.414352 24588 leader_election.cc:302] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:57.408041 24470 raft_consensus.cc:2454] T 4e5332a5d329493991dff994e70ea92c P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 6.
I20260925 03:04:57.409981 24546 raft_consensus.cc:2454] T 0b5df303cd6a4d59aca5c18bbbff24ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 6.
W20260925 03:04:57.411670 24585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.4:43979, user_credentials={real_user=slave}} blocked reactor thread for 54877us
I20260925 03:04:57.417562 24588 leader_election.cc:302] T a7aa47c48404465b94a36fd819fe3c7a P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:57.418143 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f1fac028a144058b92331c0d9b44ebf" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:57.418838 24546 raft_consensus.cc:2454] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 5.
I20260925 03:04:57.417615 24467 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1177) took 1436 ms. Trace:
I20260925 03:04:57.420475 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e54ca1757ee4e42956a9cebbeca4f62" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:57.420090 24467 rpcz_store.cc:276] 0925 03:04:55.981090 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:56.085672 (+104582us) service_pool.cc:227] Handling call
0925 03:04:56.098461 (+ 12789us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:56.947468 (+849007us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:57.417399 (+469931us) spinlock_profiling.cc:243] Waited 497 ms on lock 0x7b54002c1354. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d0f51 00007f43764cf99e 00007f437b4112e9 00007f4374af7684 00007f4374af7612 00007f4374af7541 00007f4374af74bc 00007f4374af6772 00007f43748fa94c 00007f43748f9d86
0925 03:04:57.417561 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1142222080}
I20260925 03:04:57.432323 26846 consensus_queue.cc:1035] T 37a157996e944db2906110d87ecb0c54 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.011s
I20260925 03:04:57.402020 26864 raft_consensus.cc:482] T 25102ba08e4e49388f6176badcd10a25 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:57.445232 24588 leader_election.cc:302] T 0b5df303cd6a4d59aca5c18bbbff24ed P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:57.401378 26937 raft_consensus.cc:482] T a4abe0ea757e4886be4945c5001e529a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:57.406369 26945 raft_consensus.cc:482] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:57.446363 26799 leader_election.cc:288] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:57.424153 25193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1e85248d3b04b979dd080caeeb02fdb" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:57.420023 26909 raft_consensus.cc:2781] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260925 03:04:57.619704 24470 raft_consensus.cc:1261] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260925 03:04:57.659802 24615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9c33fd100ff44c1a3247319a324a044" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:57.641481 24467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e54ca1757ee4e42956a9cebbeca4f62" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:57.500774 24616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
W20260925 03:04:57.524955 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 79793.8us
I20260925 03:04:57.536172 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abd5b8dc1b7e49e3aa9af900d73ff0e7" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:57.577481 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5332a5d329493991dff994e70ea92c" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:57.526140 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5aec38e66a947c681e6c2a72b05e6ca" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
W20260925 03:04:57.610241 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 363521us
I20260925 03:04:57.660717 24615 raft_consensus.cc:2454] T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 6.
I20260925 03:04:57.522929 26864 raft_consensus.cc:504] T 25102ba08e4e49388f6176badcd10a25 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:57.424162 26508 consensus_queue.cc:1035] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.242s
I20260925 03:04:57.453653 26756 raft_consensus.cc:2781] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260925 03:04:57.578203 26829 consensus_meta.cc:287] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c: Time spent flushing consensus metadata: real 0.624s user 0.007s sys 0.003s
I20260925 03:04:57.521844 26378 raft_consensus.cc:2781] T a7aa47c48404465b94a36fd819fe3c7a P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260925 03:04:57.585719 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7aa47c48404465b94a36fd819fe3c7a" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
W20260925 03:04:57.568825 26942 log.cc:924] Time spent T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5: Append to log took a long time: real 0.260s user 0.001s sys 0.000s
I20260925 03:04:57.573699 26577 raft_consensus.cc:3032] T 3d89396576884f7c8901dba54f783947 P eeab93abfdb141a895002ec9f312296c [term 3 LEADER]: Stepping down as leader of term 3
I20260925 03:04:57.677752 24436 leader_election.cc:302] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:57.552635 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e725b82985746b39c0cc8f671c95353" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:57.627862 25191 raft_consensus.cc:1261] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43 [term 5 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 5 index: 8. (index mismatch)
I20260925 03:04:57.625195 26874 raft_consensus.cc:482] T ceab6b0fccd54e909a64f66fe81a9a1f P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:57.685197 26897 raft_consensus.cc:2781] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260925 03:04:57.631116 26946 raft_consensus.cc:482] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:57.554965 26842 raft_consensus.cc:2781] T 0b5df303cd6a4d59aca5c18bbbff24ed P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260925 03:04:57.582469 26938 leader_election.cc:288] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:57.719538 26842 raft_consensus.cc:482] T 0b5df303cd6a4d59aca5c18bbbff24ed P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:57.719954 26842 raft_consensus.cc:3037] T 0b5df303cd6a4d59aca5c18bbbff24ed P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Advancing to term 8
I20260925 03:04:57.688793 26762 leader_election.cc:288] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:57.682600 26577 raft_consensus.cc:729] T 3d89396576884f7c8901dba54f783947 P eeab93abfdb141a895002ec9f312296c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:57.721563 26577 consensus_queue.cc:260] T 3d89396576884f7c8901dba54f783947 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:04:57.692219 26943 log.cc:924] Time spent T 550118bc1a0641ce9803bdbf43f2069f P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.109s user 0.000s sys 0.001s
I20260925 03:04:57.693208 24436 leader_election.cc:302] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: 1bc1f2e62ab44af995165bd5116a9d43, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:57.624625 26882 raft_consensus.cc:2781] T c88cbd39e01f44628a094af516be2833 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260925 03:04:57.629236 26883 raft_consensus.cc:482] T 25102ba08e4e49388f6176badcd10a25 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:57.667923 26756 raft_consensus.cc:482] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:57.662998 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1e85248d3b04b979dd080caeeb02fdb" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:57.662338 24613 raft_consensus.cc:2454] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 8.
I20260925 03:04:57.663834 24614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f1fac028a144058b92331c0d9b44ebf" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
W20260925 03:04:57.663021 24514 leader_election.cc:334] T b4ebd2b043024a548005c8b6b00b0449 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187): Timed out: RequestConsensusVote RPC to 127.22.193.1:42187 timed out after 2.117s (SENT)
I20260925 03:04:57.668991 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:57.733444 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abd5b8dc1b7e49e3aa9af900d73ff0e7" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:57.733700 24470 raft_consensus.cc:2454] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 7.
I20260925 03:04:57.734169 24620 raft_consensus.cc:2454] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 5.
I20260925 03:04:57.681684 26923 raft_consensus.cc:482] T 04e068a0f80941dc9138cee063f53bab P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:57.682355 24547 raft_consensus.cc:2454] T a7aa47c48404465b94a36fd819fe3c7a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 4.
I20260925 03:04:57.669970 24467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abd5b8dc1b7e49e3aa9af900d73ff0e7" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:57.665120 24618 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "abd5b8dc1b7e49e3aa9af900d73ff0e7"
dest_uuid: "eeab93abfdb141a895002ec9f312296c"
mode: GRACEFUL
new_leader_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
from {username='slave'} at 127.0.0.1:44570
I20260925 03:04:57.662484 24546 raft_consensus.cc:2454] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 5.
I20260925 03:04:57.738415 24618 raft_consensus.cc:595] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Received request to transfer leadership to TS a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:57.662141 24616 raft_consensus.cc:2379] T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1bc1f2e62ab44af995165bd5116a9d43 in current term 7: Already voted for candidate eeab93abfdb141a895002ec9f312296c in this term.
I20260925 03:04:57.738857 24618 raft_consensus.cc:603] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260925 03:04:57.662896 26949 log.cc:924] Time spent T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5: Append to log took a long time: real 0.078s user 0.001s sys 0.000s
I20260925 03:04:57.727001 26946 raft_consensus.cc:504] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:57.751121 26946 leader_election.cc:288] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:57.751255 24464 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1186) took 1407 ms. Trace:
I20260925 03:04:57.751513 26762 raft_consensus.cc:482] T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:57.751720 24464 rpcz_store.cc:276] 0925 03:04:56.343665 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:56.791749 (+448084us) service_pool.cc:227] Handling call
0925 03:04:56.901487 (+109738us) raft_consensus.cc:1457] Updating replica for 3 ops
0925 03:04:57.089190 (+187703us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:57.089222 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 3 ops
0925 03:04:57.101107 (+ 11885us) log.cc:842] Serialized 78 byte log entry
0925 03:04:57.276218 (+175111us) raft_consensus.cc:1631] Marking committed up to 8
0925 03:04:57.324064 (+ 47846us) log.cc:842] Serialized 14 byte log entry
0925 03:04:57.327856 (+ 3792us) log.cc:842] Serialized 14 byte log entry
0925 03:04:57.374122 (+ 46266us) log.cc:842] Serialized 14 byte log entry
0925 03:04:57.624274 (+250152us) spinlock_profiling.cc:243] Waited 7.2 ms on lock 0x7b18003b1ac0. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f437c4478b1 00007f437c454661 00007f437351f4ca 00007f437cc00f99 00007f43733e3990 00007f43735a5864 00007f43735a5155 00007f43764c83d0 00007f43764da576 00007f43764eb0d7 00007f43764eb058
0925 03:04:57.672439 (+ 48165us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:57.710810 (+ 38371us) spinlock_profiling.cc:243] Waited 178 ms on lock 0x7b54002c0954. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d0f51 00007f43764cf99e 00007f437b4112e9 00007f4374af7684 00007f4374af7612 00007f4374af7541 00007f4374af74bc 00007f4374af6772 00007f43748fa94c 00007f43748f9d86
0925 03:04:57.710866 (+ 56us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:04:57.750998 (+ 40132us) raft_consensus.cc:1671] finished
0925 03:04:57.751030 (+ 32us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:57.751185 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":297375,"prepare.queue_time_us":684507,"prepare.run_cpu_time_us":76,"prepare.run_wall_time_us":78,"raft.queue_time_us":382886,"raft.run_cpu_time_us":362,"raft.run_wall_time_us":10426,"spinlock_wait_cycles":426357248,"thread_start_us":140411,"threads_started":1,"wal-append.queue_time_us":299475}
I20260925 03:04:57.708549 26913 raft_consensus.cc:504] T 06f4a7eaf6534e4986c9fd35911038e4 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:57.755098 26803 raft_consensus.cc:504] T 8ac466793d094325ace224708b2158ed P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:57.725371 26874 raft_consensus.cc:504] T ceab6b0fccd54e909a64f66fe81a9a1f P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:57.756839 26913 leader_election.cc:288] T 06f4a7eaf6534e4986c9fd35911038e4 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 8 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:57.732985 26909 raft_consensus.cc:482] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:57.757712 26909 raft_consensus.cc:3037] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Advancing to term 9
I20260925 03:04:57.666392 26864 leader_election.cc:288] T 25102ba08e4e49388f6176badcd10a25 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:57.702121 26799 raft_consensus.cc:482] T 29637f5a4a504ce6ba29033ccd0cd716 P 1bc1f2e62ab44af995165bd5116a9d43 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:57.664852 26807 leader_election.cc:288] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:57.669049 26378 raft_consensus.cc:482] T a7aa47c48404465b94a36fd819fe3c7a P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:57.668853 26682 leader_election.cc:288] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:57.671864 26937 raft_consensus.cc:504] T a4abe0ea757e4886be4945c5001e529a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:57.705410 26863 consensus_queue.cc:1035] T cc2df40db08649c9a75e90000b878559 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260925 03:04:57.705579 26696 consensus_queue.cc:1035] T edcf6e6a422d4d198d4fbb0f7d3d845d P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260925 03:04:57.706254 26828 consensus_queue.cc:1035] T 71f461a6e2ff434993b78a2d077f0abd P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20260925 03:04:57.398670 24472 rpcz_store.cc:276] 0925 03:04:56.342931 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:56.459365 (+116434us) service_pool.cc:227] Handling call
0925 03:04:56.548390 (+ 89025us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:04:56.548824 (+ 434us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:56.548864 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:04:56.949525 (+400661us) log.cc:842] Serialized 26 byte log entry
0925 03:04:57.272334 (+322809us) raft_consensus.cc:1631] Marking committed up to 4
0925 03:04:57.285054 (+ 12720us) log.cc:842] Serialized 14 byte log entry
0925 03:04:57.285242 (+ 188us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:57.285319 (+ 77us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:04:57.398071 (+112752us) raft_consensus.cc:1671] finished
0925 03:04:57.398092 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:57.398235 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4879,"prepare.queue_time_us":51362,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":40,"raft.queue_time_us":15117,"raft.run_cpu_time_us":262,"raft.run_wall_time_us":38418,"thread_start_us":230751,"threads_started":1,"wal-append.queue_time_us":431173}
I20260925 03:04:57.708891 26659 consensus_queue.cc:1035] T 367277daf8d04778b3fa4ed94d71580d P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260925 03:04:57.644773 26900 leader_election.cc:288] T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:57.714733 24543 raft_consensus.cc:1261] T 1414dcfe53f64a78ae413531eb8ec117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260925 03:04:57.715801 26831 consensus_meta.cc:287] T 367277daf8d04778b3fa4ed94d71580d P 1bc1f2e62ab44af995165bd5116a9d43: Time spent flushing consensus metadata: real 0.634s user 0.007s sys 0.000s
I20260925 03:04:57.662211 24541 raft_consensus.cc:2454] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 5.
I20260925 03:04:57.735111 24470 raft_consensus.cc:1261] T cc2df40db08649c9a75e90000b878559 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260925 03:04:57.735898 24436 leader_election.cc:302] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:57.662209 24588 leader_election.cc:302] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:57.738371 24467 raft_consensus.cc:2454] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 5.
I20260925 03:04:57.736836 26756 raft_consensus.cc:3037] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:04:57.746621 26861 leader_election.cc:288] T b627ecb4a5384ef392aa695e06599c39 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:57.729386 26843 raft_consensus.cc:2726] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260925 03:04:57.718307 26897 raft_consensus.cc:482] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:57.747406 26947 consensus_queue.cc:1035] T 293ef69caf514ac98510a68495f54eda P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260925 03:04:57.748415 26758 leader_election.cc:288] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:57.661458 26945 raft_consensus.cc:504] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:57.736093 26882 raft_consensus.cc:482] T c88cbd39e01f44628a094af516be2833 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:57.762132 24471 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1187) took 1418 ms. Trace:
I20260925 03:04:57.763309 26378 raft_consensus.cc:3037] T a7aa47c48404465b94a36fd819fe3c7a P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:04:57.759639 26923 raft_consensus.cc:504] T 04e068a0f80941dc9138cee063f53bab P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:57.829262 26577 raft_consensus.cc:3037] T 3d89396576884f7c8901dba54f783947 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Advancing to term 4
W20260925 03:04:57.764460 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 101463us
I20260925 03:04:57.831060 26923 leader_election.cc:288] T 04e068a0f80941dc9138cee063f53bab P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:57.767320 25162 leader_election.cc:302] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:04:57.803949 26900 raft_consensus.cc:2781] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260925 03:04:57.799983 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 137865us
W20260925 03:04:57.799000 24437 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.513s user 0.031s sys 0.029s
W20260925 03:04:57.840610 24437 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.554s user 0.032s sys 0.029s
I20260925 03:04:57.803197 24472 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c85f5d1d3f004cf5817bb594fd8e6469"
dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
mode: GRACEFUL
new_leader_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
from {username='slave'} at 127.0.0.1:38832
W20260925 03:04:57.797516 26754 consensus_meta.cc:287] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c: Time spent flushing consensus metadata: real 0.500s user 0.009s sys 0.000s
I20260925 03:04:57.807112 24467 raft_consensus.cc:1226] T 06f4a7eaf6534e4986c9fd35911038e4 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 7. Current term is 8. Ops: []
I20260925 03:04:57.757103 26883 raft_consensus.cc:504] T 25102ba08e4e49388f6176badcd10a25 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:57.784528 26874 leader_election.cc:288] T ceab6b0fccd54e909a64f66fe81a9a1f P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:57.802068 26804 raft_consensus.cc:482] T 367277daf8d04778b3fa4ed94d71580d P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:57.825150 24471 rpcz_store.cc:276] 0925 03:04:56.343884 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:56.877040 (+533156us) service_pool.cc:227] Handling call
0925 03:04:56.901546 (+ 24506us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:04:56.901962 (+ 416us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:56.901992 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:04:56.922414 (+ 20422us) log.cc:842] Serialized 26 byte log entry
0925 03:04:57.144915 (+222501us) raft_consensus.cc:1631] Marking committed up to 3
0925 03:04:57.433109 (+288194us) log.cc:842] Serialized 14 byte log entry
0925 03:04:57.629648 (+196539us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:57.629722 (+ 74us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:04:57.761886 (+132164us) raft_consensus.cc:1671] finished
0925 03:04:57.761905 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:57.762076 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":211242,"prepare.queue_time_us":37915,"prepare.run_cpu_time_us":214,"prepare.run_wall_time_us":214,"raft.queue_time_us":214951,"raft.run_cpu_time_us":215,"raft.run_wall_time_us":216,"spinlock_wait_cycles":43392,"thread_start_us":484730,"threads_started":2,"wal-append.queue_time_us":322971}
I20260925 03:04:57.833194 26882 raft_consensus.cc:3037] T c88cbd39e01f44628a094af516be2833 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:04:57.668502 26829 raft_consensus.cc:504] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:57.874961 26829 leader_election.cc:288] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 6 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:57.834158 24464 raft_consensus.cc:1226] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Rejecting Update request from peer 1bc1f2e62ab44af995165bd5116a9d43 for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20260925 03:04:57.841902 26803 leader_election.cc:288] T 8ac466793d094325ace224708b2158ed P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 6 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
W20260925 03:04:57.798080 24436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 62266.8us
I20260925 03:04:57.752100 26762 raft_consensus.cc:504] T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:57.803413 26897 raft_consensus.cc:3037] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Advancing to term 7
W20260925 03:04:57.839278 25162 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 72061.6us
I20260925 03:04:57.803185 26682 raft_consensus.cc:2781] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260925 03:04:57.842303 24437 leader_election.cc:302] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:57.841946 24472 raft_consensus.cc:595] T c85f5d1d3f004cf5817bb594fd8e6469 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Received request to transfer leadership to TS a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:57.887080 24472 raft_consensus.cc:603] T c85f5d1d3f004cf5817bb594fd8e6469 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260925 03:04:57.890592 24618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35992316cb8d4c028333388eff02ecde" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:57.891146 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e068a0f80941dc9138cee063f53bab" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:57.892163 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b627ecb4a5384ef392aa695e06599c39" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:57.895125 24617 raft_consensus.cc:2354] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 5. Current term is 6.
I20260925 03:04:57.886251 24436 leader_election.cc:302] T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: 1bc1f2e62ab44af995165bd5116a9d43, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:57.899295 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceab6b0fccd54e909a64f66fe81a9a1f" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
W20260925 03:04:57.843995 24545 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.505s user 0.014s sys 0.004s
I20260925 03:04:57.763963 26799 raft_consensus.cc:504] T 29637f5a4a504ce6ba29033ccd0cd716 P 1bc1f2e62ab44af995165bd5116a9d43 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:57.900175 24616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06f4a7eaf6534e4986c9fd35911038e4" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
W20260925 03:04:57.900418 24545 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.562s user 0.014s sys 0.004s
W20260925 03:04:57.831228 24514 leader_election.cc:334] T 11bb46bce3564841b7674e6a693bd2e6 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187): Timed out: RequestConsensusVote RPC to 127.22.193.1:42187 timed out after 2.132s (SENT)
I20260925 03:04:57.900908 24514 leader_election.cc:302] T 11bb46bce3564841b7674e6a693bd2e6 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
W20260925 03:04:57.901815 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 70643.2us
I20260925 03:04:57.842216 26900 raft_consensus.cc:482] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260925 03:04:57.902050 24438 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.501s user 0.016s sys 0.023s
W20260925 03:04:57.902449 24438 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.612s user 0.016s sys 0.023s
I20260925 03:04:57.858615 26659 consensus_queue.cc:1035] T 367277daf8d04778b3fa4ed94d71580d P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260925 03:04:57.902678 26811 raft_consensus.cc:2726] T 11bb46bce3564841b7674e6a693bd2e6 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260925 03:04:57.859184 26824 leader_election.cc:288] T 21c43af5e6844fdaacc8c625fa01fd8d P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:57.843571 26749 consensus_queue.cc:1035] T 21c43af5e6844fdaacc8c625fa01fd8d P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.016s
I20260925 03:04:57.859910 26738 raft_consensus.cc:3037] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:04:57.893868 26758 raft_consensus.cc:482] T 442d4a29b1d743599f4b6b4871e26668 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:57.860720 26937 leader_election.cc:288] T a4abe0ea757e4886be4945c5001e529a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:57.863095 26883 leader_election.cc:288] T 25102ba08e4e49388f6176badcd10a25 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:57.860533 26804 raft_consensus.cc:504] T 367277daf8d04778b3fa4ed94d71580d P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:57.904596 26758 raft_consensus.cc:504] T 442d4a29b1d743599f4b6b4871e26668 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:04:57.905030 24514 leader_election.cc:334] T 442d4a29b1d743599f4b6b4871e26668 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187): Timed out: RequestConsensusVote RPC to 127.22.193.1:42187 timed out after 2.150s (SENT)
I20260925 03:04:57.905449 24514 leader_election.cc:302] T 442d4a29b1d743599f4b6b4871e26668 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:57.865692 26842 raft_consensus.cc:504] T 0b5df303cd6a4d59aca5c18bbbff24ed P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:57.844010 26843 raft_consensus.cc:482] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:57.907612 26843 raft_consensus.cc:504] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:57.867188 24588 leader_election.cc:302] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:57.868650 26509 consensus_queue.cc:1035] T cc2df40db08649c9a75e90000b878559 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260925 03:04:57.908785 26900 raft_consensus.cc:3037] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:04:57.909771 26509 raft_consensus.cc:2781] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260925 03:04:57.894106 26874 raft_consensus.cc:482] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:57.910117 26937 raft_consensus.cc:2726] T 442d4a29b1d743599f4b6b4871e26668 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260925 03:04:57.910113 26861 raft_consensus.cc:2726] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260925 03:04:57.910547 26874 raft_consensus.cc:504] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:57.911821 26842 leader_election.cc:288] T 0b5df303cd6a4d59aca5c18bbbff24ed P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 8 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:57.911841 26799 leader_election.cc:288] T 29637f5a4a504ce6ba29033ccd0cd716 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:57.912674 25192 raft_consensus.cc:1261] T 367277daf8d04778b3fa4ed94d71580d P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 6. (term mismatch)
I20260925 03:04:57.913075 25192 pending_rounds.cc:76] T 367277daf8d04778b3fa4ed94d71580d P 1bc1f2e62ab44af995165bd5116a9d43: Aborting all ops after (but not including) 5
I20260925 03:04:57.913285 25192 pending_rounds.cc:97] T 367277daf8d04778b3fa4ed94d71580d P 1bc1f2e62ab44af995165bd5116a9d43: Aborting uncommitted NO_OP operation due to leader change: 5.6
I20260925 03:04:57.913563 25192 raft_consensus.cc:2866] T 367277daf8d04778b3fa4ed94d71580d P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260925 03:04:57.915941 26804 leader_election.cc:288] T 367277daf8d04778b3fa4ed94d71580d P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:57.899986 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21c43af5e6844fdaacc8c625fa01fd8d" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:57.848865 26754 raft_consensus.cc:504] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:04:57.920912 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 79396.2us
I20260925 03:04:57.900758 24616 raft_consensus.cc:3037] T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Advancing to term 8
W20260925 03:04:57.821975 24511 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.514s user 0.007s sys 0.011s
W20260925 03:04:57.924605 24511 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.617s user 0.007s sys 0.012s
W20260925 03:04:57.925467 24511 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.4:43979, user_credentials={real_user=slave}} blocked reactor thread for 617792us
W20260925 03:04:57.913511 24514 leader_election.cc:334] T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187): Timed out: RequestConsensusVote RPC to 127.22.193.1:42187 timed out after 2.239s (SENT)
I20260925 03:04:57.927924 24514 leader_election.cc:302] T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:57.928731 24437 leader_election.cc:398] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 pre-election: Vote denied by peer a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389) with higher term. Message: Invalid argument: T 367277daf8d04778b3fa4ed94d71580d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 5. Current term is 6.
I20260925 03:04:57.933284 24511 leader_election.cc:398] T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Vote denied by peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105) with higher term. Message: Invalid argument: T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 5. Current term is 6.
I20260925 03:04:57.933647 24511 leader_election.cc:401] T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20260925 03:04:57.945020 24437 leader_election.cc:302] T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:57.945436 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21c43af5e6844fdaacc8c625fa01fd8d" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:57.946337 24437 leader_election.cc:302] T a7aa47c48404465b94a36fd819fe3c7a P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
W20260925 03:04:57.913430 26293 consensus_meta.cc:287] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c: Time spent flushing consensus metadata: real 0.588s user 0.009s sys 0.000s
I20260925 03:04:57.920077 26900 raft_consensus.cc:504] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:57.953596 24472 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:55552 (request call id 448) took 1597 ms. Trace:
I20260925 03:04:57.955399 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25102ba08e4e49388f6176badcd10a25" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:57.956044 24472 rpcz_store.cc:276] 0925 03:04:56.355949 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:57.898141 (+1542192us) service_pool.cc:227] Handling call
0925 03:04:57.953526 (+ 55385us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:57.956656 24622 raft_consensus.cc:2396] T 25102ba08e4e49388f6176badcd10a25 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260925 03:04:57.953924 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ac466793d094325ace224708b2158ed" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:57.894520 26803 raft_consensus.cc:482] T e6b6ec010caf4689b8c90449764f5cf9 P 1bc1f2e62ab44af995165bd5116a9d43 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:57.894754 26682 raft_consensus.cc:482] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:57.957667 24618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60d45b7d467d4e1ab92367f6d7d52b15" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:57.910156 26509 raft_consensus.cc:482] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:57.962708 24437 leader_election.cc:398] T f22ddd7d1c444dc58f45a1fb0b3e02bc P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 election: Vote denied by peer a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389) with higher term. Message: Invalid argument: T f22ddd7d1c444dc58f45a1fb0b3e02bc P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 6. Current term is 7.
I20260925 03:04:57.963608 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:57.972476 24437 leader_election.cc:401] T f22ddd7d1c444dc58f45a1fb0b3e02bc P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20260925 03:04:57.968103 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c81a1b3b616492ab6c964f45af36077" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
W20260925 03:04:57.969247 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 55755.9us
I20260925 03:04:57.969874 25199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4abe0ea757e4886be4945c5001e529a" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:57.943480 26861 raft_consensus.cc:482] T 8e5819c15eac4544a57e5ee6c9d4f117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:57.977099 26861 raft_consensus.cc:504] T 8e5819c15eac4544a57e5ee6c9d4f117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:57.977371 26803 raft_consensus.cc:504] T e6b6ec010caf4689b8c90449764f5cf9 P 1bc1f2e62ab44af995165bd5116a9d43 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:57.979302 26861 leader_election.cc:288] T 8e5819c15eac4544a57e5ee6c9d4f117 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:57.979353 26803 leader_election.cc:288] T e6b6ec010caf4689b8c90449764f5cf9 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:57.969098 26682 raft_consensus.cc:3037] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:04:57.955674 26946 raft_consensus.cc:2726] T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260925 03:04:57.981202 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367277daf8d04778b3fa4ed94d71580d" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:57.982434 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367277daf8d04778b3fa4ed94d71580d" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:57.951925 26947 raft_consensus.cc:482] T e6b6ec010caf4689b8c90449764f5cf9 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:57.984891 26947 raft_consensus.cc:504] T e6b6ec010caf4689b8c90449764f5cf9 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:57.985558 26913 raft_consensus.cc:2726] T a7aa47c48404465b94a36fd819fe3c7a P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260925 03:04:57.986304 26863 consensus_queue.cc:1035] T 367277daf8d04778b3fa4ed94d71580d P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260925 03:04:57.986652 26378 raft_consensus.cc:504] T a7aa47c48404465b94a36fd819fe3c7a P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:57.987355 26947 leader_election.cc:288] T e6b6ec010caf4689b8c90449764f5cf9 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:57.987653 26509 raft_consensus.cc:3037] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:04:57.988396 26945 leader_election.cc:288] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:57.988725 26843 leader_election.cc:288] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:57.936080 26811 raft_consensus.cc:482] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:57.977137 24622 raft_consensus.cc:1261] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260925 03:04:58.007045 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abd5b8dc1b7e49e3aa9af900d73ff0e7" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:58.015316 24616 raft_consensus.cc:2454] T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 8.
I20260925 03:04:58.019320 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e5819c15eac4544a57e5ee6c9d4f117" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:58.042337 24472 raft_consensus.cc:2454] T 8e5819c15eac4544a57e5ee6c9d4f117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 6.
I20260925 03:04:58.043510 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b5df303cd6a4d59aca5c18bbbff24ed" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:58.044106 24472 raft_consensus.cc:3037] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Advancing to term 8
I20260925 03:04:58.017848 26938 raft_consensus.cc:3037] T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Advancing to term 6
I20260925 03:04:58.029520 25191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6b6ec010caf4689b8c90449764f5cf9" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:58.030678 26810 raft_consensus.cc:482] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:58.031035 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4abe0ea757e4886be4945c5001e529a" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:58.055357 24588 leader_election.cc:302] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:58.056197 24616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5f9b503b0ab43999d582015a367441e" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:58.056854 24616 raft_consensus.cc:2454] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 7.
I20260925 03:04:58.056875 24436 leader_election.cc:302] T 06f4a7eaf6534e4986c9fd35911038e4 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:58.031663 26758 leader_election.cc:288] T 442d4a29b1d743599f4b6b4871e26668 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:57.995198 26900 leader_election.cc:288] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 7 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:58.031960 25198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c81a1b3b616492ab6c964f45af36077" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:58.032193 26807 raft_consensus.cc:482] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:57.961053 26293 raft_consensus.cc:504] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:58.020067 26811 raft_consensus.cc:504] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:58.016963 25192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e92f377a79354a6ebb4eec67303c13e3" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43"
I20260925 03:04:58.026615 26874 leader_election.cc:288] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
W20260925 03:04:58.037858 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 75301.6us
I20260925 03:04:58.042944 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e5819c15eac4544a57e5ee6c9d4f117" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:58.062242 24617 raft_consensus.cc:1226] T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 7. Current term is 8. Ops: []
I20260925 03:04:58.062915 24436 leader_election.cc:302] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
W20260925 03:04:58.107049 24436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 44235.3us
I20260925 03:04:58.107347 24437 leader_election.cc:302] T ac9357d144aa49789faaf31fe5e6bf89 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:58.045023 26861 raft_consensus.cc:2726] T f22ddd7d1c444dc58f45a1fb0b3e02bc P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389) with higher term. Message: Invalid argument: T f22ddd7d1c444dc58f45a1fb0b3e02bc P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 6. Current term is 7.
I20260925 03:04:58.109774 24436 leader_election.cc:302] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: 1bc1f2e62ab44af995165bd5116a9d43, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:58.112830 25198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367277daf8d04778b3fa4ed94d71580d" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:58.121724 24436 leader_election.cc:302] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: 1bc1f2e62ab44af995165bd5116a9d43, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:58.040977 26843 raft_consensus.cc:482] T 0be4e0aa984f4f2b81a58e382e434ad5 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:58.073390 26878 raft_consensus.cc:2781] T 06f4a7eaf6534e4986c9fd35911038e4 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Leader election won for term 8
I20260925 03:04:58.044793 26682 raft_consensus.cc:504] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:58.124989 24618 raft_consensus.cc:2354] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 7. Current term is 8.
I20260925 03:04:58.102911 26738 raft_consensus.cc:2781] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260925 03:04:58.128485 24437 leader_election.cc:398] T 7b8e0d260a884ecdb20f79de46c36b19 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 election: Vote denied by peer a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389) with higher term. Message: Invalid argument: T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 5. Current term is 6.
I20260925 03:04:58.131161 24437 leader_election.cc:401] T 7b8e0d260a884ecdb20f79de46c36b19 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20260925 03:04:58.113888 25192 raft_consensus.cc:3037] T e92f377a79354a6ebb4eec67303c13e3 P 1bc1f2e62ab44af995165bd5116a9d43 [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:04:58.111616 26946 raft_consensus.cc:2781] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260925 03:04:58.122814 26874 raft_consensus.cc:2726] T ac9357d144aa49789faaf31fe5e6bf89 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260925 03:04:58.127537 26913 raft_consensus.cc:2726] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260925 03:04:58.074025 25191 raft_consensus.cc:2454] T e6b6ec010caf4689b8c90449764f5cf9 P 1bc1f2e62ab44af995165bd5116a9d43 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 5.
W20260925 03:04:58.135536 24466 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.761s user 0.009s sys 0.051s
W20260925 03:04:58.154421 24466 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.780s user 0.009s sys 0.051s
I20260925 03:04:58.152890 24437 leader_election.cc:302] T 4e5332a5d329493991dff994e70ea92c P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:58.130457 26738 raft_consensus.cc:482] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:58.107182 26807 raft_consensus.cc:504] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:58.069705 26810 raft_consensus.cc:504] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:58.138463 26878 raft_consensus.cc:686] T 06f4a7eaf6534e4986c9fd35911038e4 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 LEADER]: Becoming Leader. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:04:58.130923 26843 raft_consensus.cc:504] T 0be4e0aa984f4f2b81a58e382e434ad5 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:04:58.234791 25196 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.569s user 0.001s sys 0.021s
W20260925 03:04:58.235769 25196 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.570s user 0.001s sys 0.021s
I20260925 03:04:58.214799 24436 leader_election.cc:302] T e6b6ec010caf4689b8c90449764f5cf9 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5; no voters:
I20260925 03:04:58.161398 24472 raft_consensus.cc:2454] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 8.
I20260925 03:04:58.242691 24588 leader_election.cc:302] T 0b5df303cd6a4d59aca5c18bbbff24ed P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:58.145376 26946 raft_consensus.cc:482] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260925 03:04:58.165872 24539 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.775s user 0.024s sys 0.065s
I20260925 03:04:58.220805 26878 consensus_queue.cc:237] T 06f4a7eaf6534e4986c9fd35911038e4 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:04:58.215847 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 64193us
I20260925 03:04:58.262332 24437 leader_election.cc:398] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 election: Vote denied by peer a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389) with higher term. Message: Invalid argument: T b030d84c670b48d5bff337ccc0c77626 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 7. Current term is 8.
I20260925 03:04:58.262647 24437 leader_election.cc:401] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 election: Cancelling election due to peer responding with higher term
W20260925 03:04:58.216400 24473 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.590s user 0.000s sys 0.019s
W20260925 03:04:58.263257 24473 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.637s user 0.000s sys 0.020s
W20260925 03:04:58.216663 24469 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.810s user 0.003s sys 0.027s
W20260925 03:04:58.264139 24469 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.858s user 0.003s sys 0.027s
W20260925 03:04:58.157258 24468 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.740s user 0.006s sys 0.025s
W20260925 03:04:58.264866 24468 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.848s user 0.006s sys 0.025s
I20260925 03:04:58.244208 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367277daf8d04778b3fa4ed94d71580d" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:58.267103 24472 raft_consensus.cc:2454] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 6.
W20260925 03:04:58.160449 25194 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.808s user 0.009s sys 0.009s
I20260925 03:04:58.273233 24437 leader_election.cc:398] T 37a157996e944db2906110d87ecb0c54 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 election: Vote denied by peer a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389) with higher term. Message: Invalid argument: T 37a157996e944db2906110d87ecb0c54 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 6. Current term is 7.
I20260925 03:04:58.273595 24437 leader_election.cc:401] T 37a157996e944db2906110d87ecb0c54 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20260925 03:04:58.275823 26843 leader_election.cc:288] T 0be4e0aa984f4f2b81a58e382e434ad5 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
W20260925 03:04:58.208372 24548 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.589s user 0.019s sys 0.025s
W20260925 03:04:58.258510 24539 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.868s user 0.024s sys 0.069s
W20260925 03:04:58.280714 24548 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.662s user 0.020s sys 0.025s
W20260925 03:04:58.282035 25194 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.919s user 0.009s sys 0.013s
I20260925 03:04:58.282449 26810 leader_election.cc:288] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:58.156545 26945 consensus_queue.cc:1035] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.130s
I20260925 03:04:58.259838 26923 raft_consensus.cc:2781] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260925 03:04:58.298847 26945 raft_consensus.cc:2726] T 37a157996e944db2906110d87ecb0c54 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389) with higher term. Message: Invalid argument: T 37a157996e944db2906110d87ecb0c54 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 6. Current term is 7.
I20260925 03:04:58.300472 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5f9b503b0ab43999d582015a367441e" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:58.269381 26758 raft_consensus.cc:2781] T 0b5df303cd6a4d59aca5c18bbbff24ed P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Leader election won for term 8
I20260925 03:04:58.269508 26378 leader_election.cc:288] T a7aa47c48404465b94a36fd819fe3c7a P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 6 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:58.165477 26909 raft_consensus.cc:504] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:58.270161 26874 raft_consensus.cc:2726] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Leader election lost for term 7. Reason: Vote denied by peer a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389) with higher term. Message: Invalid argument: T b030d84c670b48d5bff337ccc0c77626 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 7. Current term is 8.
I20260925 03:04:58.269610 26947 raft_consensus.cc:2726] T 7b8e0d260a884ecdb20f79de46c36b19 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389) with higher term. Message: Invalid argument: T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 5. Current term is 6.
I20260925 03:04:58.305307 24615 raft_consensus.cc:1226] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 8. Current term is 9. Ops: [8.9-8.9]
I20260925 03:04:58.221926 26807 leader_election.cc:288] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:58.310070 24615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be4e0aa984f4f2b81a58e382e434ad5" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:58.157596 26738 raft_consensus.cc:3037] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:04:58.257156 26824 raft_consensus.cc:2726] T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260925 03:04:58.234244 26956 raft_consensus.cc:482] T cc2df40db08649c9a75e90000b878559 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:58.233820 26913 raft_consensus.cc:2726] T 4e5332a5d329493991dff994e70ea92c P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260925 03:04:58.321919 24620 raft_consensus.cc:2454] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 6.
W20260925 03:04:58.332122 24544 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.653s user 0.008s sys 0.025s
W20260925 03:04:58.332454 24544 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.653s user 0.008s sys 0.025s
I20260925 03:04:58.269127 26946 raft_consensus.cc:3037] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Advancing to term 8
W20260925 03:04:58.321548 26646 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.661s user 0.008s sys 0.029s
I20260925 03:04:58.333031 25162 leader_election.cc:302] T 8ac466793d094325ace224708b2158ed P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:58.221875 26897 raft_consensus.cc:504] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:04:58.333117 24540 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.619s user 0.014s sys 0.014s
I20260925 03:04:58.150894 26882 raft_consensus.cc:504] T c88cbd39e01f44628a094af516be2833 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:04:58.326565 26820 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.659s user 0.008s sys 0.024s
I20260925 03:04:58.298873 26923 raft_consensus.cc:482] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:58.270536 26954 raft_consensus.cc:2781] T e6b6ec010caf4689b8c90449764f5cf9 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260925 03:04:58.334903 24586 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.627s user 0.017s sys 0.010s
W20260925 03:04:58.338874 24540 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.625s user 0.014s sys 0.014s
W20260925 03:04:58.342001 24586 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.635s user 0.018s sys 0.011s
W20260925 03:04:58.342382 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 635050us
I20260925 03:04:58.330260 26758 raft_consensus.cc:686] T 0b5df303cd6a4d59aca5c18bbbff24ed P eeab93abfdb141a895002ec9f312296c [term 8 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
W20260925 03:04:58.337188 26646 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.683s user 0.008s sys 0.030s
I20260925 03:04:58.340979 26803 raft_consensus.cc:2781] T 8ac466793d094325ace224708b2158ed P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 FOLLOWER]: Leader election won for term 6
W20260925 03:04:58.349346 24465 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.897s user 0.014s sys 0.027s
I20260925 03:04:58.323590 26874 consensus_queue.cc:1046] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.022s
W20260925 03:04:58.350857 24465 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.898s user 0.014s sys 0.028s
I20260925 03:04:58.343283 26293 leader_election.cc:288] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 8 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:58.353511 26803 raft_consensus.cc:686] T 8ac466793d094325ace224708b2158ed P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 LEADER]: Becoming Leader. State: Replica: 1bc1f2e62ab44af995165bd5116a9d43, State: Running, Role: LEADER
W20260925 03:04:58.357498 24470 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.515s user 0.009s sys 0.019s
I20260925 03:04:58.342805 26954 raft_consensus.cc:482] T e6b6ec010caf4689b8c90449764f5cf9 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:58.344851 26923 raft_consensus.cc:3037] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [term 8 FOLLOWER]: Advancing to term 9
I20260925 03:04:58.357322 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abd5b8dc1b7e49e3aa9af900d73ff0e7" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:58.355753 24615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c88cbd39e01f44628a094af516be2833" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:58.360716 24615 raft_consensus.cc:3037] T c88cbd39e01f44628a094af516be2833 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:04:58.341285 26509 raft_consensus.cc:504] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:04:58.348765 26820 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.680s user 0.009s sys 0.024s
W20260925 03:04:58.369479 24470 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.527s user 0.011s sys 0.020s
I20260925 03:04:58.356181 26937 raft_consensus.cc:2726] T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 LEADER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260925 03:04:58.376657 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5f9b503b0ab43999d582015a367441e" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:58.377395 24541 raft_consensus.cc:2454] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 6.
I20260925 03:04:58.337852 26956 raft_consensus.cc:504] T cc2df40db08649c9a75e90000b878559 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:58.358639 26861 raft_consensus.cc:2726] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260925 03:04:58.357729 26682 leader_election.cc:288] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:58.382072 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9c33fd100ff44c1a3247319a324a044" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:58.383100 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9c33fd100ff44c1a3247319a324a044" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:58.358549 26954 raft_consensus.cc:3037] T e6b6ec010caf4689b8c90449764f5cf9 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:04:58.383525 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e5819c15eac4544a57e5ee6c9d4f117" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:58.356205 26882 leader_election.cc:288] T c88cbd39e01f44628a094af516be2833 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
W20260925 03:04:58.391814 24619 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.679s user 0.019s sys 0.015s
I20260925 03:04:58.386410 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25102ba08e4e49388f6176badcd10a25" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:58.387338 24545 raft_consensus.cc:2454] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 5.
I20260925 03:04:58.392484 24542 raft_consensus.cc:2454] T 25102ba08e4e49388f6176badcd10a25 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 3.
I20260925 03:04:58.393175 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e92f377a79354a6ebb4eec67303c13e3" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:58.388643 24541 raft_consensus.cc:2454] T 8e5819c15eac4544a57e5ee6c9d4f117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 5.
I20260925 03:04:58.388106 24620 raft_consensus.cc:2454] T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 6.
I20260925 03:04:58.384883 24613 raft_consensus.cc:2379] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 6: Already voted for candidate eeab93abfdb141a895002ec9f312296c in this term.
W20260925 03:04:58.395643 24619 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.717s user 0.019s sys 0.015s
I20260925 03:04:58.374607 26754 leader_election.cc:288] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 7 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:58.384907 24511 leader_election.cc:398] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 7 pre-election: Vote denied by peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105) with higher term. Message: Invalid argument: T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 7. Current term is 8.
I20260925 03:04:58.409574 24466 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1200) took 1434 ms. Trace:
I20260925 03:04:58.375945 26762 leader_election.cc:288] T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:58.413738 25197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15861cd28e354598a3c6bfc4337c34bb" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:58.414628 25197 raft_consensus.cc:2379] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 5: Already voted for candidate 1bc1f2e62ab44af995165bd5116a9d43 in this term.
I20260925 03:04:58.362762 26293 consensus_queue.cc:1035] T 1414dcfe53f64a78ae413531eb8ec117 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.031s
I20260925 03:04:58.384158 26800 consensus_queue.cc:1035] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.023s
I20260925 03:04:58.411015 24511 leader_election.cc:401] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 7 pre-election: Cancelling election due to peer responding with higher term
W20260925 03:04:58.443710 24511 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 58945.3us
I20260925 03:04:58.412843 24466 rpcz_store.cc:276] 0925 03:04:56.975274 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:57.288821 (+313547us) service_pool.cc:227] Handling call
0925 03:04:57.328777 (+ 39956us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:57.328928 (+ 151us) raft_consensus.cc:1520] Early marking committed up to index 5
0925 03:04:58.154873 (+825945us) log.cc:842] Serialized 14 byte log entry
0925 03:04:58.408506 (+253633us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:58.409332 (+ 826us) raft_consensus.cc:1631] Marking committed up to 5
0925 03:04:58.409375 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:58.409439 (+ 64us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:58.409524 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":252319,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":42,"raft.queue_time_us":6756,"thread_start_us":1025874,"threads_started":2}
I20260925 03:04:58.458163 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b5df303cd6a4d59aca5c18bbbff24ed" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:58.458922 24543 raft_consensus.cc:3037] T 0b5df303cd6a4d59aca5c18bbbff24ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Advancing to term 8
I20260925 03:04:58.392496 26810 consensus_queue.cc:1046] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.060s
I20260925 03:04:58.466209 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25102ba08e4e49388f6176badcd10a25" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:58.466898 24466 raft_consensus.cc:2454] T 25102ba08e4e49388f6176badcd10a25 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 4.
I20260925 03:04:58.468016 24586 leader_election.cc:302] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:58.468099 24588 leader_election.cc:302] T 25102ba08e4e49388f6176badcd10a25 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:58.470328 24586 leader_election.cc:302] T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:58.471433 25198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:58.428113 26811 leader_election.cc:288] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:58.472087 25198 raft_consensus.cc:2454] T 7e49248643df4131a1c42e74244e05c2 P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 6.
I20260925 03:04:58.452451 26955 raft_consensus.cc:482] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:58.392974 26758 consensus_queue.cc:237] T 0b5df303cd6a4d59aca5c18bbbff24ed P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:04:58.481002 24514 connection.cc:232] RPC call timeout handler was delayed by -1.4795186729988927s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21930000000000002s for a response
W20260925 03:04:58.491709 24514 connection.cc:232] RPC call timeout handler was delayed by -1.4783865610006615s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1757s for a response
W20260925 03:04:58.492072 24514 connection.cc:232] RPC call timeout handler was delayed by -1.4521385589996498s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2942s for a response
W20260925 03:04:58.492555 24514 leader_election.cc:334] T b030d84c670b48d5bff337ccc0c77626 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187): Timed out: RequestConsensusVote RPC to 127.22.193.1:42187 timed out after 1.530s (SENT)
I20260925 03:04:58.492933 24514 leader_election.cc:302] T b030d84c670b48d5bff337ccc0c77626 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:58.482542 24465 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1203) took 1437 ms. Trace:
I20260925 03:04:58.488795 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9c33fd100ff44c1a3247319a324a044" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:58.495352 24466 raft_consensus.cc:2379] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c in current term 7: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:04:58.493844 26811 raft_consensus.cc:3037] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Advancing to term 8
I20260925 03:04:58.500394 26509 leader_election.cc:288] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 6 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
W20260925 03:04:58.503144 26964 log.cc:924] Time spent T 7b8e0d260a884ecdb20f79de46c36b19 P 869b0a8575b54d08ae2817fad5dd42d5: Append to log took a long time: real 0.077s user 0.000s sys 0.000s
I20260925 03:04:58.504827 24473 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1202) took 1460 ms. Trace:
I20260925 03:04:58.505170 24473 rpcz_store.cc:276] 0925 03:04:57.044805 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:57.387973 (+343168us) service_pool.cc:227] Handling call
0925 03:04:57.388280 (+ 307us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:04:57.625141 (+236861us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:57.625178 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:04:57.625704 (+ 526us) log.cc:842] Serialized 26 byte log entry
0925 03:04:58.263644 (+637940us) raft_consensus.cc:1631] Marking committed up to 3
0925 03:04:58.263689 (+ 45us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:58.263767 (+ 78us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:04:58.504601 (+240834us) raft_consensus.cc:1671] finished
0925 03:04:58.504622 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:58.504773 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":236305,"raft.queue_time_us":240523,"raft.run_cpu_time_us":53,"raft.run_wall_time_us":52,"thread_start_us":637550,"threads_started":1,"wal-append.queue_time_us":764425}
I20260925 03:04:58.506762 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6b6ec010caf4689b8c90449764f5cf9" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:58.507472 24615 raft_consensus.cc:2454] T c88cbd39e01f44628a094af516be2833 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 7.
W20260925 03:04:58.508597 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 235491us
I20260925 03:04:58.477046 26712 raft_consensus.cc:2781] T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260925 03:04:58.509506 24437 leader_election.cc:398] T b4ebd2b043024a548005c8b6b00b0449 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 pre-election: Vote denied by peer a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389) with higher term. Message: Invalid argument: T b4ebd2b043024a548005c8b6b00b0449 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 3. Current term is 4.
I20260925 03:04:58.475567 26378 raft_consensus.cc:2781] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260925 03:04:58.509819 24437 leader_election.cc:401] T b4ebd2b043024a548005c8b6b00b0449 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20260925 03:04:58.481655 26749 raft_consensus.cc:2781] T 25102ba08e4e49388f6176badcd10a25 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260925 03:04:58.510373 26749 raft_consensus.cc:482] T 25102ba08e4e49388f6176badcd10a25 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:58.510480 26843 raft_consensus.cc:2726] T b4ebd2b043024a548005c8b6b00b0449 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389) with higher term. Message: Invalid argument: T b4ebd2b043024a548005c8b6b00b0449 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 3. Current term is 4.
I20260925 03:04:58.510747 26749 raft_consensus.cc:3037] T 25102ba08e4e49388f6176badcd10a25 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:58.511708 24437 leader_election.cc:398] T edcf6e6a422d4d198d4fbb0f7d3d845d P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 election: Vote denied by peer a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389) with higher term. Message: Invalid argument: T edcf6e6a422d4d198d4fbb0f7d3d845d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 4. Current term is 5.
I20260925 03:04:58.511972 24437 leader_election.cc:401] T edcf6e6a422d4d198d4fbb0f7d3d845d P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20260925 03:04:58.512481 26843 raft_consensus.cc:2726] T edcf6e6a422d4d198d4fbb0f7d3d845d P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389) with higher term. Message: Invalid argument: T edcf6e6a422d4d198d4fbb0f7d3d845d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 4. Current term is 5.
I20260925 03:04:58.513096 26829 raft_consensus.cc:979] T a4abe0ea757e4886be4945c5001e529a P eeab93abfdb141a895002ec9f312296c: : Instructing follower a04a37fc4aa7460cbfdbeeb4e451e1f5 to start an election
I20260925 03:04:58.513401 26810 raft_consensus.cc:3032] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 LEADER]: Stepping down as leader of term 7
I20260925 03:04:58.513588 26829 raft_consensus.cc:1067] T a4abe0ea757e4886be4945c5001e529a P eeab93abfdb141a895002ec9f312296c [term 2 LEADER]: Signalling peer a04a37fc4aa7460cbfdbeeb4e451e1f5 to start an election
I20260925 03:04:58.513803 26810 raft_consensus.cc:729] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:04:58.478595 26897 leader_election.cc:288] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:58.514783 24437 leader_election.cc:302] T 7993f9c3e2a44f408bbf5d1e395e2306 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:58.514603 26810 consensus_queue.cc:260] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:58.515611 26897 raft_consensus.cc:2726] T 7993f9c3e2a44f408bbf5d1e395e2306 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260925 03:04:58.515843 26810 raft_consensus.cc:3037] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Advancing to term 8
I20260925 03:04:58.516669 24437 leader_election.cc:302] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:04:58.509611 26712 raft_consensus.cc:482] T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:58.517935 26712 raft_consensus.cc:3037] T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:04:58.518388 24544 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 1111) took 1217 ms. Trace:
I20260925 03:04:58.518730 24544 rpcz_store.cc:276] 0925 03:04:57.301265 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:57.304094 (+ 2829us) service_pool.cc:227] Handling call
0925 03:04:57.329961 (+ 25867us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:04:57.341676 (+ 11715us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:57.341706 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:04:57.342139 (+ 433us) log.cc:842] Serialized 26 byte log entry
0925 03:04:57.640669 (+298530us) spinlock_profiling.cc:243] Waited 10.5 ms on lock 0x7b180001c2a0. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f437c4478b1 00007f437c454661 00007f437358a345 00007f437c3c1a39 00007f43735a78bb 00007f43735a5cbf 00007f43735a7e22 00007f43761515bb 00007f4376157063 00007f4376157411 00007f43764a45cf
0925 03:04:58.332809 (+692140us) raft_consensus.cc:1631] Marking committed up to 3
0925 03:04:58.332848 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:58.332914 (+ 66us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:04:58.518097 (+185183us) raft_consensus.cc:1671] finished
0925 03:04:58.518114 (+ 17us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:58.518331 (+ 217us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":148,"raft.queue_time_us":37742,"raft.run_cpu_time_us":81,"raft.run_wall_time_us":81,"spinlock_wait_cycles":24056192,"thread_start_us":653235,"threads_started":1,"wal-append.queue_time_us":1032323}
I20260925 03:04:58.491971 26955 raft_consensus.cc:504] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:58.523012 26955 leader_election.cc:288] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:58.523649 24470 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1212) took 1192 ms. Trace:
I20260925 03:04:58.524013 24470 rpcz_store.cc:276] 0925 03:04:57.331439 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:57.800949 (+469510us) service_pool.cc:227] Handling call
0925 03:04:57.801269 (+ 320us) raft_consensus.cc:1457] Updating replica for 2 ops
0925 03:04:57.823601 (+ 22332us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:57.823639 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0925 03:04:57.824226 (+ 587us) log.cc:842] Serialized 52 byte log entry
0925 03:04:58.370857 (+546631us) raft_consensus.cc:1631] Marking committed up to 1
0925 03:04:58.370901 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:58.370988 (+ 87us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:04:58.523438 (+152450us) raft_consensus.cc:1671] finished
0925 03:04:58.523459 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:58.523593 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1286,"raft.queue_time_us":21075,"raft.run_cpu_time_us":122,"raft.run_wall_time_us":10220,"thread_start_us":527833,"threads_started":1,"wal-append.queue_time_us":608305}
W20260925 03:04:58.525782 26958 log.cc:924] Time spent T 71f461a6e2ff434993b78a2d077f0abd P 1bc1f2e62ab44af995165bd5116a9d43: Append to log took a long time: real 0.167s user 0.002s sys 0.000s
I20260925 03:04:58.525715 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7aa47c48404465b94a36fd819fe3c7a" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:58.526324 24470 raft_consensus.cc:3037] T a7aa47c48404465b94a36fd819fe3c7a P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Advancing to term 6
W20260925 03:04:58.529291 26961 log.cc:924] Time spent T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5: Append to log took a long time: real 0.122s user 0.000s sys 0.000s
I20260925 03:04:58.530071 25192 raft_consensus.cc:2454] T e92f377a79354a6ebb4eec67303c13e3 P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 6.
I20260925 03:04:58.531533 24469 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1199) took 1556 ms. Trace:
I20260925 03:04:58.532570 26909 leader_election.cc:288] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 9 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:58.533077 26293 raft_consensus.cc:482] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:58.533538 26293 raft_consensus.cc:504] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:58.531881 24469 rpcz_store.cc:276] 0925 03:04:56.975021 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:57.268192 (+293171us) service_pool.cc:227] Handling call
0925 03:04:57.289101 (+ 20909us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:04:57.289314 (+ 213us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:57.289339 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:04:57.315465 (+ 26126us) log.cc:842] Serialized 26 byte log entry
0925 03:04:58.264531 (+949066us) raft_consensus.cc:1631] Marking committed up to 3
0925 03:04:58.264572 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:58.264641 (+ 69us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:04:58.531337 (+266696us) raft_consensus.cc:1671] finished
0925 03:04:58.531358 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:58.531478 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":858219,"threads_started":1,"wal-append.queue_time_us":1014434}
W20260925 03:04:58.536098 24514 leader_election.cc:334] T edcf6e6a422d4d198d4fbb0f7d3d845d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187): Timed out: RequestConsensusVote RPC to 127.22.193.1:42187 timed out after 1.581s (SENT)
I20260925 03:04:58.536535 24514 leader_election.cc:302] T edcf6e6a422d4d198d4fbb0f7d3d845d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:58.537876 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "293ef69caf514ac98510a68495f54eda" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:58.538537 24544 raft_consensus.cc:2454] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 5.
I20260925 03:04:58.539522 24436 leader_election.cc:302] T c88cbd39e01f44628a094af516be2833 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:58.542096 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9c33fd100ff44c1a3247319a324a044" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
W20260925 03:04:58.521463 26960 log.cc:924] Time spent T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.161s user 0.000s sys 0.000s
I20260925 03:04:58.547453 24470 raft_consensus.cc:2454] T a7aa47c48404465b94a36fd819fe3c7a P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 6.
I20260925 03:04:58.548171 24585 leader_election.cc:302] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, eeab93abfdb141a895002ec9f312296c; no voters:
W20260925 03:04:58.507565 26756 consensus_meta.cc:287] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c: Time spent flushing consensus metadata: real 0.681s user 0.009s sys 0.000s
I20260925 03:04:58.566335 24620 raft_consensus.cc:1226] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 8. Current term is 9. Ops: [8.9-8.9]
I20260925 03:04:58.568970 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:58.570569 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5278a60f97da4e10aecd50a8d9a9a43c" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:58.570647 24543 raft_consensus.cc:2454] T 0b5df303cd6a4d59aca5c18bbbff24ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 8.
I20260925 03:04:58.558115 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e54ca1757ee4e42956a9cebbeca4f62" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:58.564088 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5332a5d329493991dff994e70ea92c" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:58.550215 26979 raft_consensus.cc:2726] T b030d84c670b48d5bff337ccc0c77626 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260925 03:04:58.541478 26378 raft_consensus.cc:482] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:58.577275 26829 raft_consensus.cc:2781] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Leader election won for term 6
I20260925 03:04:58.565268 26682 raft_consensus.cc:2781] T c88cbd39e01f44628a094af516be2833 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader election won for term 7
I20260925 03:04:58.590559 26829 raft_consensus.cc:686] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [term 6 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:58.591485 26829 consensus_queue.cc:237] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:58.593048 24588 leader_election.cc:302] T a7aa47c48404465b94a36fd819fe3c7a P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:58.593318 26923 raft_consensus.cc:504] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:58.594396 26846 raft_consensus.cc:2781] T a7aa47c48404465b94a36fd819fe3c7a P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Leader election won for term 6
I20260925 03:04:58.594925 26846 raft_consensus.cc:686] T a7aa47c48404465b94a36fd819fe3c7a P eeab93abfdb141a895002ec9f312296c [term 6 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:58.595767 26846 consensus_queue.cc:237] T a7aa47c48404465b94a36fd819fe3c7a P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:58.461742 26803 consensus_queue.cc:237] T 8ac466793d094325ace224708b2158ed P 1bc1f2e62ab44af995165bd5116a9d43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:58.590408 26946 raft_consensus.cc:504] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:58.578226 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:58.547709 26897 raft_consensus.cc:2781] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260925 03:04:58.584025 26756 raft_consensus.cc:504] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:58.614444 26682 raft_consensus.cc:686] T c88cbd39e01f44628a094af516be2833 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 LEADER]: Becoming Leader. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:04:58.582263 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4abe0ea757e4886be4945c5001e529a" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:58.617259 24614 raft_consensus.cc:2379] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 in current term 7: Already voted for candidate eeab93abfdb141a895002ec9f312296c in this term.
I20260925 03:04:58.579895 24437 leader_election.cc:302] T 1414dcfe53f64a78ae413531eb8ec117 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:04:58.609625 24543 raft_consensus.cc:2379] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1bc1f2e62ab44af995165bd5116a9d43 in current term 7: Already voted for candidate eeab93abfdb141a895002ec9f312296c in this term.
W20260925 03:04:58.571246 26938 consensus_meta.cc:287] T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent flushing consensus metadata: real 0.509s user 0.010s sys 0.000s
I20260925 03:04:58.613669 24539 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 1104) took 1314 ms. Trace:
I20260925 03:04:58.578852 24466 raft_consensus.cc:3037] T 4e5332a5d329493991dff994e70ea92c P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:04:58.615060 24472 raft_consensus.cc:2379] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c in current term 8: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:04:58.578480 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c23337a7cb74184a0e57b7095cb6a11" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
W20260925 03:04:58.581394 26965 log.cc:924] Time spent T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43: Append to log took a long time: real 0.130s user 0.000s sys 0.000s
W20260925 03:04:58.581877 26976 log.cc:924] Time spent T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5: Append to log took a long time: real 0.156s user 0.000s sys 0.000s
I20260925 03:04:58.639034 24468 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1182) took 2296 ms. Trace:
I20260925 03:04:58.582753 26378 raft_consensus.cc:3037] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Advancing to term 8
I20260925 03:04:58.639225 26682 consensus_queue.cc:237] T c88cbd39e01f44628a094af516be2833 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:58.613242 26897 raft_consensus.cc:482] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260925 03:04:58.620039 26983 log.cc:924] Time spent T b627ecb4a5384ef392aa695e06599c39 P 869b0a8575b54d08ae2817fad5dd42d5: Append to log took a long time: real 0.104s user 0.000s sys 0.001s
I20260925 03:04:58.623533 24539 rpcz_store.cc:276] 0925 03:04:57.299318 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:57.301368 (+ 2050us) service_pool.cc:227] Handling call
0925 03:04:57.306778 (+ 5410us) raft_consensus.cc:1457] Updating replica for 2 ops
0925 03:04:57.307137 (+ 359us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:57.307166 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0925 03:04:57.307734 (+ 568us) log.cc:842] Serialized 52 byte log entry
0925 03:04:58.280457 (+972723us) raft_consensus.cc:1631] Marking committed up to 4
0925 03:04:58.280500 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:58.280567 (+ 67us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:04:58.613455 (+332888us) raft_consensus.cc:1671] finished
0925 03:04:58.613476 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:58.613614 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1097,"raft.run_cpu_time_us":184,"raft.run_wall_time_us":20691,"thread_start_us":889701,"threads_started":1,"wal-append.queue_time_us":1013313}
I20260925 03:04:58.623998 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5f9b503b0ab43999d582015a367441e" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:58.620878 26874 consensus_queue.cc:1046] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.018s
I20260925 03:04:58.639496 25196 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48408 (request call id 206) took 1001 ms. Trace:
I20260925 03:04:58.624660 24470 raft_consensus.cc:3037] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:04:58.639533 24468 rpcz_store.cc:276] 0925 03:04:56.342662 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:56.458780 (+116118us) service_pool.cc:227] Handling call
0925 03:04:56.901756 (+442976us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:04:57.179666 (+277910us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:57.179707 (+ 41us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:04:57.224408 (+ 44701us) log.cc:842] Serialized 26 byte log entry
0925 03:04:57.396261 (+171853us) raft_consensus.cc:1631] Marking committed up to 3
0925 03:04:58.265232 (+868971us) log.cc:842] Serialized 14 byte log entry
0925 03:04:58.537730 (+272498us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:58.622721 (+ 84991us) spinlock_profiling.cc:243] Waited 266 ms on lock 0x7b540040d3d4. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d0f51 00007f43764cf99e 00007f437b4112e9 00007f4374af7684 00007f4374af7612 00007f4374af7541 00007f4374af74bc 00007f4374af6772 00007f43748fa94c 00007f43748f9d86
0925 03:04:58.622772 (+ 51us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:04:58.622819 (+ 47us) raft_consensus.cc:1671] finished
0925 03:04:58.638792 (+ 15973us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:58.638962 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":41773,"prepare.queue_time_us":185151,"prepare.run_cpu_time_us":27,"prepare.run_wall_time_us":27,"raft.queue_time_us":126773,"raft.run_cpu_time_us":276,"raft.run_wall_time_us":53279,"spinlock_wait_cycles":612352384,"thread_start_us":1245120,"threads_started":3,"wal-append.queue_time_us":582580}
I20260925 03:04:58.620142 26956 leader_election.cc:288] T cc2df40db08649c9a75e90000b878559 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:58.609355 25192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e54ca1757ee4e42956a9cebbeca4f62" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43"
I20260925 03:04:58.653210 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367277daf8d04778b3fa4ed94d71580d" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:58.654227 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e54ca1757ee4e42956a9cebbeca4f62" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:58.655435 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60d45b7d467d4e1ab92367f6d7d52b15" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:58.655977 24472 raft_consensus.cc:3037] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Advancing to term 8
I20260925 03:04:58.656806 24466 raft_consensus.cc:2454] T 4e5332a5d329493991dff994e70ea92c P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 7.
I20260925 03:04:58.657680 26984 raft_consensus.cc:2741] T 1414dcfe53f64a78ae413531eb8ec117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader election decision vote started in defunct term 3: won
I20260925 03:04:58.627990 26938 raft_consensus.cc:2726] T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105) with higher term. Message: Invalid argument: T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 5. Current term is 6.
I20260925 03:04:58.661777 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f1fac028a144058b92331c0d9b44ebf" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:58.640333 26897 raft_consensus.cc:3037] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Advancing to term 8
I20260925 03:04:58.599638 25194 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49794 (request call id 250) took 1249 ms. Trace:
I20260925 03:04:58.586404 26712 raft_consensus.cc:504] T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:58.619542 26756 leader_election.cc:288] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 7 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:58.642668 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:58.665789 24464 raft_consensus.cc:2454] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 7.
W20260925 03:04:58.615295 26978 log.cc:924] Time spent T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43: Append to log took a long time: real 0.116s user 0.000s sys 0.001s
I20260925 03:04:58.620718 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e068a0f80941dc9138cee063f53bab" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:58.667738 24545 raft_consensus.cc:2454] T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 6.
I20260925 03:04:58.577848 26293 leader_election.cc:288] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:58.662444 24468 raft_consensus.cc:3037] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Advancing to term 7
W20260925 03:04:58.662760 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 82956.2us
I20260925 03:04:58.663076 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abd5b8dc1b7e49e3aa9af900d73ff0e7" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:58.649495 25196 rpcz_store.cc:276] 0925 03:04:57.638181 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:57.638362 (+ 181us) service_pool.cc:227] Handling call
0925 03:04:57.638783 (+ 421us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:04:57.665036 (+ 26253us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:57.665065 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:04:57.665544 (+ 479us) log.cc:842] Serialized 26 byte log entry
0925 03:04:58.236802 (+571258us) raft_consensus.cc:1631] Marking committed up to 8
0925 03:04:58.236846 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:58.236924 (+ 78us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:04:58.639268 (+402344us) raft_consensus.cc:1671] finished
0925 03:04:58.639291 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:58.639437 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":25862,"raft.queue_time_us":27139,"raft.run_cpu_time_us":93,"raft.run_wall_time_us":93,"thread_start_us":570870,"threads_started":1,"wal-append.queue_time_us":718730}
I20260925 03:04:58.669965 24588 leader_election.cc:302] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:58.662165 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5332a5d329493991dff994e70ea92c" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:58.670795 24539 raft_consensus.cc:3037] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Advancing to term 7
I20260925 03:04:58.669967 24437 leader_election.cc:302] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:58.598766 26749 raft_consensus.cc:504] T 25102ba08e4e49388f6176badcd10a25 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:58.619477 26946 leader_election.cc:288] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 8 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:58.668236 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e54ca1757ee4e42956a9cebbeca4f62" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:58.696178 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7aa47c48404465b94a36fd819fe3c7a" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:58.697368 24543 raft_consensus.cc:3037] T a7aa47c48404465b94a36fd819fe3c7a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Advancing to term 6
I20260925 03:04:58.697363 24613 raft_consensus.cc:2379] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 1bc1f2e62ab44af995165bd5116a9d43 in current term 9: Already voted for candidate eeab93abfdb141a895002ec9f312296c in this term.
I20260925 03:04:58.700438 24614 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38570 (request call id 1537) took 1037 ms. Trace:
I20260925 03:04:58.701203 24588 leader_election.cc:302] T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:58.701663 24622 raft_consensus.cc:2379] T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 in current term 7: Already voted for candidate eeab93abfdb141a895002ec9f312296c in this term.
I20260925 03:04:58.685257 24470 raft_consensus.cc:2454] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 7.
I20260925 03:04:58.686944 24585 leader_election.cc:302] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:04:58.696121 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f68804eb5a244d87bb395d50683c6012" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:58.702581 24464 raft_consensus.cc:2454] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 6.
I20260925 03:04:58.671849 24466 raft_consensus.cc:2379] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c in current term 6: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:04:58.703287 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60d45b7d467d4e1ab92367f6d7d52b15" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
W20260925 03:04:58.705739 24511 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.4:43979, user_credentials={real_user=slave}} blocked reactor thread for 261349us
I20260925 03:04:58.701607 24614 rpcz_store.cc:276] 0925 03:04:57.663188 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:57.663601 (+ 413us) service_pool.cc:227] Handling call
0925 03:04:58.700242 (+1036641us) spinlock_profiling.cc:243] Waited 882 ms on lock 0x7b5400073fd4. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d42ae 00007f437b411c27 00007f4374af9c04 00007f4374af9b92 00007f4374af9ac1 00007f4374af9a3c 00007f4374af8cf2 00007f43748fa94c 00007f43748f9d86 00007f43748f972f
0925 03:04:58.700369 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2028271360}
I20260925 03:04:58.710568 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f1fac028a144058b92331c0d9b44ebf" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:58.712987 24544 raft_consensus.cc:3037] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Advancing to term 7
I20260925 03:04:58.715819 24588 leader_election.cc:302] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:58.713958 24472 raft_consensus.cc:2454] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 8.
I20260925 03:04:58.718302 25162 leader_election.cc:302] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:58.719300 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15861cd28e354598a3c6bfc4337c34bb" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:58.665693 26923 leader_election.cc:288] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 9 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:58.723969 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21c43af5e6844fdaacc8c625fa01fd8d" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:58.724609 24464 raft_consensus.cc:2454] T 21c43af5e6844fdaacc8c625fa01fd8d P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 4.
I20260925 03:04:58.720113 24472 raft_consensus.cc:2454] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 5.
I20260925 03:04:58.712674 24511 leader_election.cc:302] T a4abe0ea757e4886be4945c5001e529a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 1bc1f2e62ab44af995165bd5116a9d43, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:58.716595 24619 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34546 (request call id 160) took 1039 ms. Trace:
I20260925 03:04:58.730000 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abd5b8dc1b7e49e3aa9af900d73ff0e7" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:58.736364 24472 raft_consensus.cc:2379] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 6: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:04:58.737118 24539 raft_consensus.cc:2454] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 7.
I20260925 03:04:58.738230 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:58.738848 24539 raft_consensus.cc:2454] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 7.
I20260925 03:04:58.700271 26938 raft_consensus.cc:482] T 1414dcfe53f64a78ae413531eb8ec117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:58.737169 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f22ddd7d1c444dc58f45a1fb0b3e02bc" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:58.742957 24471 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38920 (request call id 156) took 1116 ms. Trace:
I20260925 03:04:58.700577 26883 raft_consensus.cc:2726] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260925 03:04:58.743628 24472 raft_consensus.cc:2379] T f22ddd7d1c444dc58f45a1fb0b3e02bc P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 7: Already voted for candidate eeab93abfdb141a895002ec9f312296c in this term.
I20260925 03:04:58.743419 24471 rpcz_store.cc:276] 0925 03:04:57.626126 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:57.849186 (+223060us) service_pool.cc:227] Handling call
0925 03:04:57.849538 (+ 352us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:04:57.860872 (+ 11334us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:57.860898 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:04:57.861381 (+ 483us) log.cc:842] Serialized 26 byte log entry
0925 03:04:58.360190 (+498809us) raft_consensus.cc:1631] Marking committed up to 5
0925 03:04:58.360226 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:58.360288 (+ 62us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:04:58.742758 (+382470us) raft_consensus.cc:1671] finished
0925 03:04:58.742776 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:58.742891 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":12568,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":126,"thread_start_us":498365,"threads_started":1,"wal-append.queue_time_us":619556}
I20260925 03:04:58.755503 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06f4a7eaf6534e4986c9fd35911038e4" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:04:58.711285 26804 raft_consensus.cc:482] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:58.711792 26874 raft_consensus.cc:482] T c85f5d1d3f004cf5817bb594fd8e6469 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:58.727048 26970 raft_consensus.cc:2781] T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260925 03:04:58.686092 26738 raft_consensus.cc:504] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:58.739871 26946 raft_consensus.cc:482] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:58.711575 26878 raft_consensus.cc:482] T 11bb46bce3564841b7674e6a693bd2e6 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:58.756613 24472 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "cc2df40db08649c9a75e90000b878559"
dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
mode: GRACEFUL
new_leader_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
from {username='slave'} at 127.0.0.1:38832
I20260925 03:04:58.788580 24472 raft_consensus.cc:595] T cc2df40db08649c9a75e90000b878559 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Received request to transfer leadership to TS a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:04:58.788441 26878 raft_consensus.cc:504] T 11bb46bce3564841b7674e6a693bd2e6 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:58.789067 24472 raft_consensus.cc:603] T cc2df40db08649c9a75e90000b878559 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260925 03:04:58.789412 26913 raft_consensus.cc:3032] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 LEADER]: Stepping down as leader of term 8
I20260925 03:04:58.789819 26913 raft_consensus.cc:729] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:04:58.790669 26878 leader_election.cc:288] T 11bb46bce3564841b7674e6a693bd2e6 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:58.756659 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442d4a29b1d743599f4b6b4871e26668" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:58.791401 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c88cbd39e01f44628a094af516be2833" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:58.791824 24471 raft_consensus.cc:2454] T 442d4a29b1d743599f4b6b4871e26668 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 6.
I20260925 03:04:58.792541 23300 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260925 03:04:58.752615 26985 raft_consensus.cc:482] T 43366a0a2f9d4344bf4e37ffd290ef8a P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:58.793442 26985 raft_consensus.cc:504] T 43366a0a2f9d4344bf4e37ffd290ef8a P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:58.794450 23300 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
W20260925 03:04:58.794680 26972 log.cc:924] Time spent T f22ddd7d1c444dc58f45a1fb0b3e02bc P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.295s user 0.001s sys 0.001s
I20260925 03:04:58.795177 23300 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260925 03:04:58.795482 26985 leader_election.cc:288] T 43366a0a2f9d4344bf4e37ffd290ef8a P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:58.796741 24540 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 1127) took 1361 ms. Trace:
W20260925 03:04:58.797760 26977 log.cc:924] Time spent T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5: Append to log took a long time: real 0.282s user 0.002s sys 0.000s
I20260925 03:04:58.797097 24540 rpcz_store.cc:276] 0925 03:04:57.435061 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:57.435330 (+ 269us) service_pool.cc:227] Handling call
0925 03:04:57.528808 (+ 93478us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:04:57.713448 (+184640us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:57.713479 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:04:57.713929 (+ 450us) log.cc:842] Serialized 26 byte log entry
0925 03:04:57.860781 (+146852us) spinlock_profiling.cc:243] Waited 1.33 ms on lock 0x7b48000c1ed8. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f437b3777f1 00007f437b36e8e7 00007f437b36e77a 00007f437b46295f 00007f437b462627 00007f437b476b2c 00007f437b476ad8 00007f437b476a58 00007f437b476a13 00007f437b475cbc 00007f437a3f7be6
0925 03:04:58.342307 (+481526us) raft_consensus.cc:1631] Marking committed up to 3
0925 03:04:58.342349 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:58.342414 (+ 65us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:04:58.794101 (+451687us) spinlock_profiling.cc:243] Waited 44 ms on lock 0x7b44000232cc. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f43733ecf11 00007f43734baa95 00007f43734bb4b2 00007f4376163511 00007f437615ee1a 00007f4376152bee 00007f4376151f36 00007f4376151b58 00007f437615b551 00007f437615b509 00007f437615b499
0925 03:04:58.796505 (+ 2404us) raft_consensus.cc:1671] finished
0925 03:04:58.796526 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:58.796661 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":166250,"raft.queue_time_us":170017,"raft.run_cpu_time_us":2554,"raft.run_wall_time_us":143243,"spinlock_wait_cycles":104146560,"thread_start_us":627962,"threads_started":1,"wal-append.queue_time_us":742958}
I20260925 03:04:58.798904 24467 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38920 (request call id 152) took 1436 ms. Trace:
I20260925 03:04:58.719270 26293 raft_consensus.cc:2781] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Leader election won for term 7
I20260925 03:04:58.802085 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43366a0a2f9d4344bf4e37ffd290ef8a" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:58.718520 26577 raft_consensus.cc:2726] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260925 03:04:58.742859 26956 raft_consensus.cc:2726] T a4abe0ea757e4886be4945c5001e529a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260925 03:04:58.766885 26955 raft_consensus.cc:482] T 100c113a9fd84cc8adf2ec7b33ea0093 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:58.821900 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 8 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:58.686372 26954 raft_consensus.cc:504] T e6b6ec010caf4689b8c90449764f5cf9 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:58.761914 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9c33fd100ff44c1a3247319a324a044" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:58.826427 24464 raft_consensus.cc:2354] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 6. Current term is 7.
I20260925 03:04:58.779001 24468 raft_consensus.cc:2454] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 7.
I20260925 03:04:58.753355 24543 raft_consensus.cc:2454] T a7aa47c48404465b94a36fd819fe3c7a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 6.
I20260925 03:04:58.766703 26820 raft_consensus.cc:2726] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260925 03:04:58.799247 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6b6ec010caf4689b8c90449764f5cf9" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:58.792539 24511 leader_election.cc:302] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:58.766938 26863 raft_consensus.cc:482] T 04e068a0f80941dc9138cee063f53bab P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:58.496719 24465 rpcz_store.cc:276] 0925 03:04:57.045109 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:57.405026 (+359917us) service_pool.cc:227] Handling call
0925 03:04:57.405351 (+ 325us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:57.405557 (+ 206us) raft_consensus.cc:1520] Early marking committed up to index 3
0925 03:04:58.355366 (+949809us) log.cc:842] Serialized 14 byte log entry
0925 03:04:58.482185 (+126819us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:58.482244 (+ 59us) raft_consensus.cc:1631] Marking committed up to 3
0925 03:04:58.482290 (+ 46us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:58.482359 (+ 69us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:58.482484 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":31644,"prepare.queue_time_us":176053,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":32,"raft.queue_time_us":766,"raft.run_cpu_time_us":184,"raft.run_wall_time_us":4582,"thread_start_us":1028486,"threads_started":2}
I20260925 03:04:58.799255 24467 rpcz_store.cc:276] 0925 03:04:57.362285 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:57.844533 (+482248us) service_pool.cc:227] Handling call
0925 03:04:57.858935 (+ 14402us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:04:57.910028 (+ 51093us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:57.910056 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:04:57.910486 (+ 430us) log.cc:842] Serialized 26 byte log entry
0925 03:04:58.358287 (+447801us) raft_consensus.cc:1631] Marking committed up to 6
0925 03:04:58.358331 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:58.604677 (+246346us) spinlock_profiling.cc:243] Waited 50.8 ms on lock 0x7b5400192dd4. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d0f51 00007f43764cf99e 00007f437b4112e9 00007f4374af7684 00007f4374af7612 00007f4374af7541 00007f4374af74bc 00007f4374af6772 00007f43748fa94c 00007f43748f9d86
0925 03:04:58.604748 (+ 71us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:04:58.669789 (+ 65041us) spinlock_profiling.cc:243] Waited 27.2 ms on lock 0x7b44000232cc. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f43733ecf11 00007f43734baa95 00007f43734bb4b2 00007f4376163511 00007f437615ee1a 00007f4376152bee 00007f4376151f36 00007f4376151b58 00007f437615b551 00007f437615b509 00007f437615b499
0925 03:04:58.798713 (+128924us) raft_consensus.cc:1671] finished
0925 03:04:58.798734 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:58.798851 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":977,"raft.run_cpu_time_us":81,"raft.run_wall_time_us":81,"spinlock_wait_cycles":179409536,"thread_start_us":445572,"threads_started":1,"wal-append.queue_time_us":604725}
I20260925 03:04:58.801990 24614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bb46bce3564841b7674e6a693bd2e6" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:58.809533 26749 leader_election.cc:288] T 25102ba08e4e49388f6176badcd10a25 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:58.811319 24621 raft_consensus.cc:2379] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 in current term 7: Already voted for candidate eeab93abfdb141a895002ec9f312296c in this term.
I20260925 03:04:58.787304 26509 raft_consensus.cc:482] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:58.761341 26762 raft_consensus.cc:2781] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Leader election won for term 7
I20260925 03:04:58.801242 24545 raft_consensus.cc:2454] T c88cbd39e01f44628a094af516be2833 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 5.
I20260925 03:04:58.771952 26970 raft_consensus.cc:482] T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:58.897746 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21c43af5e6844fdaacc8c625fa01fd8d" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:58.898600 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ac466793d094325ace224708b2158ed" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:58.900138 24539 raft_consensus.cc:2379] T 8ac466793d094325ace224708b2158ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1bc1f2e62ab44af995165bd5116a9d43 in current term 6: Already voted for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in this term.
I20260925 03:04:58.855675 24544 raft_consensus.cc:2454] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 7.
W20260925 03:04:58.848467 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 310690us
I20260925 03:04:58.893078 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06f4a7eaf6534e4986c9fd35911038e4" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:58.759162 26378 raft_consensus.cc:504] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:58.862869 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5278a60f97da4e10aecd50a8d9a9a43c" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:58.759725 26938 raft_consensus.cc:504] T 1414dcfe53f64a78ae413531eb8ec117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:58.936168 24467 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1227) took 1010 ms. Trace:
I20260925 03:04:58.938230 24467 rpcz_store.cc:276] 0925 03:04:57.926103 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:58.856497 (+930394us) service_pool.cc:227] Handling call
0925 03:04:58.935620 (+ 79123us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:58.935829 (+ 209us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:58.935859 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:58.935901 (+ 42us) raft_consensus.cc:1631] Marking committed up to 4
0925 03:04:58.935934 (+ 33us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:58.935988 (+ 54us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:58.936105 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:58.937994 24464 raft_consensus.cc:1261] T 1414dcfe53f64a78ae413531eb8ec117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260925 03:04:58.941072 24464 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1224) took 1015 ms. Trace:
I20260925 03:04:58.941425 24464 rpcz_store.cc:276] 0925 03:04:57.925322 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:58.827123 (+901801us) service_pool.cc:227] Handling call
0925 03:04:58.901290 (+ 74167us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:58.940812 (+ 39522us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:58.941012 (+ 200us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260925 03:04:58.904536 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 188795us
I20260925 03:04:58.921057 24473 raft_consensus.cc:2379] T e6b6ec010caf4689b8c90449764f5cf9 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1bc1f2e62ab44af995165bd5116a9d43 in current term 6: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:04:58.901691 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c23337a7cb74184a0e57b7095cb6a11" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
W20260925 03:04:58.902328 24514 leader_election.cc:334] T c88cbd39e01f44628a094af516be2833 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187): Timed out: RequestConsensusVote RPC to 127.22.193.1:42187 timed out after 1.730s (SENT)
I20260925 03:04:58.730526 24619 rpcz_store.cc:276] 0925 03:04:57.676778 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:57.677090 (+ 312us) service_pool.cc:227] Handling call
0925 03:04:57.677367 (+ 277us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:57.677526 (+ 159us) raft_consensus.cc:1520] Early marking committed up to index 7
0925 03:04:58.396088 (+718562us) log.cc:842] Serialized 14 byte log entry
0925 03:04:58.715680 (+319592us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:58.715727 (+ 47us) raft_consensus.cc:1631] Marking committed up to 7
0925 03:04:58.715766 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:58.716428 (+ 662us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:58.716542 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2562,"prepare.queue_time_us":830273,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":37,"raft.queue_time_us":4376,"raft.run_cpu_time_us":119,"raft.run_wall_time_us":4020,"thread_start_us":1033715,"threads_started":2}
I20260925 03:04:58.942943 24514 leader_election.cc:302] T c88cbd39e01f44628a094af516be2833 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:58.945040 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:58.945691 24544 raft_consensus.cc:2454] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 7.
I20260925 03:04:58.904501 24545 raft_consensus.cc:2396] T 21c43af5e6844fdaacc8c625fa01fd8d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260925 03:04:58.942934 24539 raft_consensus.cc:3037] T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Advancing to term 7
I20260925 03:04:58.946473 24543 raft_consensus.cc:2454] T 5278a60f97da4e10aecd50a8d9a9a43c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 4.
I20260925 03:04:58.771617 26874 raft_consensus.cc:504] T c85f5d1d3f004cf5817bb594fd8e6469 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:58.942603 24588 leader_election.cc:302] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:58.828317 26646 raft_consensus.cc:482] T ac9357d144aa49789faaf31fe5e6bf89 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:58.969858 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:58.976377 24466 raft_consensus.cc:2454] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 7.
I20260925 03:04:58.828258 26293 raft_consensus.cc:686] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c [term 7 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:58.770802 26804 raft_consensus.cc:504] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:58.971496 24588 leader_election.cc:302] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
W20260925 03:04:58.976856 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 74578.7us
I20260925 03:04:58.981248 24620 raft_consensus.cc:2379] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 in current term 8: Already voted for candidate eeab93abfdb141a895002ec9f312296c in this term.
W20260925 03:04:58.984861 24514 leader_election.cc:334] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187): Timed out: RequestConsensusVote RPC to 127.22.193.1:42187 timed out after 1.773s (SENT)
I20260925 03:04:58.985255 24514 leader_election.cc:302] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:58.874673 26937 raft_consensus.cc:2781] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260925 03:04:58.994912 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35992316cb8d4c028333388eff02ecde" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:58.803660 26946 raft_consensus.cc:504] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:58.906590 26970 raft_consensus.cc:3037] T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Advancing to term 8
I20260925 03:04:58.998822 24540 raft_consensus.cc:2396] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 for term 8 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260925 03:04:59.007560 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b627ecb4a5384ef392aa695e06599c39" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:59.011869 24437 leader_election.cc:302] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:04:59.013985 24437 leader_election.cc:302] T 21c43af5e6844fdaacc8c625fa01fd8d P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:58.950340 26762 raft_consensus.cc:686] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [term 7 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
W20260925 03:04:59.020771 24514 leader_election.cc:334] T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187): Timed out: RequestConsensusVote RPC to 127.22.193.1:42187 timed out after 1.781s (SENT)
I20260925 03:04:59.017489 24466 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1235) took 1013 ms. Trace:
I20260925 03:04:59.021216 24514 leader_election.cc:302] T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:59.021299 24466 rpcz_store.cc:276] 0925 03:04:58.003826 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:58.977156 (+973330us) service_pool.cc:227] Handling call
0925 03:04:59.016975 (+ 39819us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:59.017165 (+ 190us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:59.017193 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:59.017234 (+ 41us) raft_consensus.cc:1631] Marking committed up to 3
0925 03:04:59.017265 (+ 31us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:59.017316 (+ 51us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:59.017430 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:58.948091 26947 raft_consensus.cc:482] T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:59.024396 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60d45b7d467d4e1ab92367f6d7d52b15" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
W20260925 03:04:59.022634 24514 leader_election.cc:334] T 0be4e0aa984f4f2b81a58e382e434ad5 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187): Timed out: RequestConsensusVote RPC to 127.22.193.1:42187 timed out after 1.800s (SENT)
I20260925 03:04:59.029310 24514 leader_election.cc:302] T 0be4e0aa984f4f2b81a58e382e434ad5 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:59.026814 26378 leader_election.cc:288] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 8 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:58.975041 26993 raft_consensus.cc:482] T 420163a6cdce412ebc21c25fe3835638 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:58.890326 26863 raft_consensus.cc:504] T 04e068a0f80941dc9138cee063f53bab P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:58.997076 26937 raft_consensus.cc:482] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:59.033454 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abd5b8dc1b7e49e3aa9af900d73ff0e7" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:59.042474 24544 raft_consensus.cc:3037] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:04:58.904527 26509 raft_consensus.cc:504] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:58.997783 26979 raft_consensus.cc:2726] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260925 03:04:59.035893 24514 leader_election.cc:334] T 43366a0a2f9d4344bf4e37ffd290ef8a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187): Timed out: RequestConsensusVote RPC to 127.22.193.1:42187 timed out after 1.834s (SENT)
I20260925 03:04:58.834249 26955 raft_consensus.cc:504] T 100c113a9fd84cc8adf2ec7b33ea0093 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:58.983893 26991 raft_consensus.cc:2726] T c88cbd39e01f44628a094af516be2833 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260925 03:04:59.046219 24514 leader_election.cc:302] T 43366a0a2f9d4344bf4e37ffd290ef8a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:59.020486 26754 raft_consensus.cc:2781] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260925 03:04:59.013164 26843 raft_consensus.cc:482] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:59.046202 24621 raft_consensus.cc:1226] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 8. Current term is 9. Ops: [8.9-8.9]
W20260925 03:04:59.035382 26981 log.cc:924] Time spent T 550118bc1a0641ce9803bdbf43f2069f P 869b0a8575b54d08ae2817fad5dd42d5: Append to log took a long time: real 0.518s user 0.000s sys 0.002s
I20260925 03:04:59.022815 26712 leader_election.cc:288] T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 7 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
W20260925 03:04:59.039049 26758 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.535s user 0.011s sys 0.012s
I20260925 03:04:59.053777 24545 raft_consensus.cc:2454] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 6.
I20260925 03:04:59.020200 26984 raft_consensus.cc:2781] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260925 03:04:59.041862 26987 log.cc:924] Time spent T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5: Append to log took a long time: real 0.346s user 0.001s sys 0.001s
I20260925 03:04:59.055697 24619 raft_consensus.cc:1261] T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260925 03:04:59.027779 26800 raft_consensus.cc:482] T 550118bc1a0641ce9803bdbf43f2069f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260925 03:04:59.050421 26811 consensus_meta.cc:287] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent flushing consensus metadata: real 0.550s user 0.010s sys 0.000s
I20260925 03:04:59.054414 26754 raft_consensus.cc:482] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:59.060164 26991 raft_consensus.cc:2726] T 43366a0a2f9d4344bf4e37ffd290ef8a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260925 03:04:59.063298 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bb46bce3564841b7674e6a693bd2e6" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:59.120707 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43366a0a2f9d4344bf4e37ffd290ef8a" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:59.121325 24541 raft_consensus.cc:2454] T 43366a0a2f9d4344bf4e37ffd290ef8a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 3.
I20260925 03:04:59.122113 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ac466793d094325ace224708b2158ed" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:59.122799 24541 raft_consensus.cc:2379] T 8ac466793d094325ace224708b2158ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 1bc1f2e62ab44af995165bd5116a9d43 in current term 6: Already voted for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in this term.
I20260925 03:04:59.123479 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be4e0aa984f4f2b81a58e382e434ad5" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:58.988389 26646 raft_consensus.cc:504] T ac9357d144aa49789faaf31fe5e6bf89 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:59.125362 26954 leader_election.cc:288] T e6b6ec010caf4689b8c90449764f5cf9 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:59.125739 26646 leader_election.cc:288] T ac9357d144aa49789faaf31fe5e6bf89 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:59.051394 26937 raft_consensus.cc:3037] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Advancing to term 6
W20260925 03:04:59.066027 26988 log.cc:924] Time spent T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.084s user 0.000s sys 0.000s
I20260925 03:04:59.126991 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5f9b503b0ab43999d582015a367441e" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:59.127604 24547 raft_consensus.cc:2454] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 6.
I20260925 03:04:59.127874 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6b6ec010caf4689b8c90449764f5cf9" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:59.128373 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abd5b8dc1b7e49e3aa9af900d73ff0e7" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:59.129000 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f68804eb5a244d87bb395d50683c6012" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:59.129551 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceab6b0fccd54e909a64f66fe81a9a1f" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:59.130095 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5f9b503b0ab43999d582015a367441e" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:59.130494 26863 leader_election.cc:288] T 04e068a0f80941dc9138cee063f53bab P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:59.130097 24547 raft_consensus.cc:2454] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 6.
I20260925 03:04:59.085070 26293 consensus_queue.cc:237] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:59.131757 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25102ba08e4e49388f6176badcd10a25" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:59.132437 26804 leader_election.cc:288] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:58.667136 25194 rpcz_store.cc:276] 0925 03:04:57.350578 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:57.350737 (+ 159us) service_pool.cc:227] Handling call
0925 03:04:57.351131 (+ 394us) raft_consensus.cc:1457] Updating replica for 2 ops
0925 03:04:57.351701 (+ 570us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:57.351735 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0925 03:04:57.352372 (+ 637us) log.cc:842] Serialized 52 byte log entry
0925 03:04:58.282946 (+930574us) raft_consensus.cc:1631] Marking committed up to 5
0925 03:04:58.283029 (+ 83us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:58.283160 (+ 131us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:04:58.525275 (+242115us) spinlock_profiling.cc:243] Waited 2.71 ms on lock 0x7b44000232cc. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f43733ecf11 00007f43734baa95 00007f43734bb4b2 00007f4376163511 00007f437615ee1a 00007f4376152bee 00007f4376151f36 00007f4376151b58 00007f437615b551 00007f437615b509 00007f437615b499
0925 03:04:58.535622 (+ 10347us) raft_consensus.cc:1671] finished
0925 03:04:58.541135 (+ 5513us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:58.578050 (+ 36915us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1297,"raft.run_cpu_time_us":107,"raft.run_wall_time_us":107,"spinlock_wait_cycles":6239104,"thread_start_us":930060,"threads_started":1,"wal-append.queue_time_us":995327}
I20260925 03:04:59.132303 24547 raft_consensus.cc:3037] T 25102ba08e4e49388f6176badcd10a25 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Advancing to term 5
I20260925 03:04:59.084436 26861 consensus_queue.cc:1046] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.049s
I20260925 03:04:59.084730 26829 raft_consensus.cc:2781] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Leader election won for term 8
I20260925 03:04:59.134363 26829 raft_consensus.cc:686] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c [term 8 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:59.135502 24473 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38920 (request call id 160) took 1099 ms. Trace:
I20260925 03:04:59.135183 26829 consensus_queue.cc:237] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:59.135867 24473 rpcz_store.cc:276] 0925 03:04:58.036477 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:58.506508 (+470031us) service_pool.cc:227] Handling call
0925 03:04:59.135334 (+628826us) spinlock_profiling.cc:243] Waited 413 ms on lock 0x7b5400091854. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d42ae 00007f437b411c27 00007f4374af9c04 00007f4374af9b92 00007f4374af9ac1 00007f4374af9a3c 00007f4374af8cf2 00007f43748fa94c 00007f43748f9d86 00007f43748f972f
0925 03:04:59.135446 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":950110080}
I20260925 03:04:59.137265 26874 leader_election.cc:288] T c85f5d1d3f004cf5817bb594fd8e6469 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:59.101788 26984 raft_consensus.cc:482] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:59.138024 26984 raft_consensus.cc:3037] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:59.139797 26946 leader_election.cc:288] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:59.043246 26993 raft_consensus.cc:504] T 420163a6cdce412ebc21c25fe3835638 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:59.142052 26993 leader_election.cc:288] T 420163a6cdce412ebc21c25fe3835638 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:59.083698 26843 raft_consensus.cc:504] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:59.143800 26897 raft_consensus.cc:504] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:59.072131 26938 leader_election.cc:288] T 1414dcfe53f64a78ae413531eb8ec117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:59.130632 24548 raft_consensus.cc:3037] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Advancing to term 8
I20260925 03:04:59.081183 24544 raft_consensus.cc:2454] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 6.
I20260925 03:04:59.033280 26980 raft_consensus.cc:2726] T edcf6e6a422d4d198d4fbb0f7d3d845d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260925 03:04:59.038157 26947 raft_consensus.cc:504] T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:59.097793 26956 raft_consensus.cc:482] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:59.073278 26979 raft_consensus.cc:2726] T 0be4e0aa984f4f2b81a58e382e434ad5 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260925 03:04:59.084669 26758 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.580s user 0.011s sys 0.012s
I20260925 03:04:59.118495 26754 raft_consensus.cc:3037] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:04:59.194375 24469 raft_consensus.cc:2379] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1bc1f2e62ab44af995165bd5116a9d43 in current term 8: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:04:59.135196 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9c33fd100ff44c1a3247319a324a044" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:59.114281 26738 leader_election.cc:288] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 7 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:59.195537 24547 raft_consensus.cc:2454] T 25102ba08e4e49388f6176badcd10a25 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 5.
I20260925 03:04:59.197875 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e725b82985746b39c0cc8f671c95353" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:59.198722 24547 raft_consensus.cc:2396] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 for term 7 because replica has last-logged OpId of term: 6 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 2.
I20260925 03:04:59.148774 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442d4a29b1d743599f4b6b4871e26668" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:59.152837 24539 raft_consensus.cc:2454] T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 7.
I20260925 03:04:59.155360 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c85f5d1d3f004cf5817bb594fd8e6469" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:59.205221 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9c33fd100ff44c1a3247319a324a044" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
W20260925 03:04:59.129580 26975 log.cc:924] Time spent T 00000000000000000000000000000000 P 2d92cc0732c4490eb7d98d353826cce1: Append to log took a long time: real 0.630s user 0.001s sys 0.002s
I20260925 03:04:59.116170 26811 raft_consensus.cc:2726] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: Vote denied by peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105) with higher term. Message: Invalid argument: T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 7. Current term is 8.
I20260925 03:04:59.206601 24544 raft_consensus.cc:2454] T 442d4a29b1d743599f4b6b4871e26668 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 5.
I20260925 03:04:59.210981 24541 raft_consensus.cc:2454] T 0be4e0aa984f4f2b81a58e382e434ad5 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 5.
I20260925 03:04:59.210987 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abd5b8dc1b7e49e3aa9af900d73ff0e7" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:59.237707 24539 raft_consensus.cc:2379] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 in current term 6: Already voted for candidate eeab93abfdb141a895002ec9f312296c in this term.
W20260925 03:04:59.193930 27003 log.cc:924] Time spent T 37a157996e944db2906110d87ecb0c54 P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.078s user 0.001s sys 0.000s
I20260925 03:04:59.116736 26878 consensus_queue.cc:1035] T 06f4a7eaf6534e4986c9fd35911038e4 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.112s
I20260925 03:04:59.249817 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f1fac028a144058b92331c0d9b44ebf" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:59.250686 24544 raft_consensus.cc:2379] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 in current term 7: Already voted for candidate eeab93abfdb141a895002ec9f312296c in this term.
W20260925 03:04:59.254307 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 218441us
W20260925 03:04:59.216873 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 245431us
I20260925 03:04:59.238276 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367277daf8d04778b3fa4ed94d71580d" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:59.280330 24588 leader_election.cc:302] T 442d4a29b1d743599f4b6b4871e26668 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:59.238611 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c88cbd39e01f44628a094af516be2833" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:04:59.112315 26800 raft_consensus.cc:504] T 550118bc1a0641ce9803bdbf43f2069f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:59.281728 24539 raft_consensus.cc:3037] T c88cbd39e01f44628a094af516be2833 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Advancing to term 7
I20260925 03:04:59.326941 24539 raft_consensus.cc:2454] T c88cbd39e01f44628a094af516be2833 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 7.
W20260925 03:04:59.227566 25165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 470182us
W20260925 03:04:59.329311 26990 log.cc:924] Time spent T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.139s user 0.000s sys 0.000s
I20260925 03:04:59.194661 26956 raft_consensus.cc:504] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:59.209388 26955 leader_election.cc:288] T 100c113a9fd84cc8adf2ec7b33ea0093 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:59.237917 26909 raft_consensus.cc:482] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1bc1f2e62ab44af995165bd5116a9d43)
I20260925 03:04:59.147006 26762 consensus_queue.cc:237] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:59.225302 24548 raft_consensus.cc:2454] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 8.
I20260925 03:04:59.209213 26810 raft_consensus.cc:2726] T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260925 03:04:59.440891 24541 raft_consensus.cc:1261] T b4ebd2b043024a548005c8b6b00b0449 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Refusing update from remote peer 1bc1f2e62ab44af995165bd5116a9d43: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260925 03:04:59.441596 24541 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33068 (request call id 154) took 1041 ms. Trace:
I20260925 03:04:59.441903 24541 rpcz_store.cc:276] 0925 03:04:58.408773 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:59.440142 (+1031369us) service_pool.cc:227] Handling call
0925 03:04:59.440465 (+ 323us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:59.441351 (+ 886us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:59.441541 (+ 190us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260925 03:04:59.344945 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 359610us
I20260925 03:04:59.291944 24543 raft_consensus.cc:2454] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 8.
I20260925 03:04:59.444000 24469 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38920 (request call id 158) took 1521 ms. Trace:
I20260925 03:04:59.444535 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e5819c15eac4544a57e5ee6c9d4f117" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
W20260925 03:04:59.445520 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 187062us
I20260925 03:04:59.444397 24469 rpcz_store.cc:276] 0925 03:04:57.922613 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:58.535956 (+613343us) service_pool.cc:227] Handling call
0925 03:04:59.443792 (+907836us) spinlock_profiling.cc:243] Waited 624 ms on lock 0x7b54001919d4. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d42ae 00007f437b411c27 00007f4374af9c04 00007f4374af9b92 00007f4374af9ac1 00007f4374af9a3c 00007f4374af8cf2 00007f43748fa94c 00007f43748f9d86 00007f43748f972f
0925 03:04:59.443939 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1436237568}
I20260925 03:04:59.417675 24545 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 1148) took 1017 ms. Trace:
W20260925 03:04:59.344556 25163 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.602s user 0.000s sys 0.013s
W20260925 03:04:59.447729 25163 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.724s user 0.003s sys 0.015s
I20260925 03:04:59.448431 24539 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 1152) took 1032 ms. Trace:
W20260925 03:04:59.440066 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 426137us
I20260925 03:04:59.451068 24548 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 1153) took 1034 ms. Trace:
I20260925 03:04:59.451802 24437 leader_election.cc:302] T b627ecb4a5384ef392aa695e06599c39 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:59.446090 24541 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 1151) took 1034 ms. Trace:
I20260925 03:04:59.452962 24541 rpcz_store.cc:276] 0925 03:04:58.411402 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:59.442653 (+1031251us) service_pool.cc:227] Handling call
0925 03:04:59.442953 (+ 300us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:59.445712 (+ 2759us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:59.445741 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:59.445783 (+ 42us) raft_consensus.cc:1631] Marking committed up to 3
0925 03:04:59.445815 (+ 32us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:59.445922 (+ 107us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:59.446042 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:59.450791 24539 rpcz_store.cc:276] 0925 03:04:58.415719 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:59.447661 (+1031942us) service_pool.cc:227] Handling call
0925 03:04:59.447954 (+ 293us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:59.448140 (+ 186us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:59.448165 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:59.448208 (+ 43us) raft_consensus.cc:1631] Marking committed up to 3
0925 03:04:59.448240 (+ 32us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:59.448287 (+ 47us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:59.448373 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260925 03:04:59.446111 24514 leader_election.cc:334] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187): Timed out: RequestConsensusVote RPC to 127.22.193.1:42187 timed out after 1.891s (SENT)
I20260925 03:04:59.454233 24471 raft_consensus.cc:1226] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 7. Current term is 8. Ops: [7.6-7.6]
I20260925 03:04:59.455046 24541 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 1154) took 1038 ms. Trace:
I20260925 03:04:59.451424 24548 rpcz_store.cc:276] 0925 03:04:58.416313 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:59.450342 (+1034029us) service_pool.cc:227] Handling call
0925 03:04:59.450604 (+ 262us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:59.450774 (+ 170us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:59.450801 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:59.450840 (+ 39us) raft_consensus.cc:1631] Marking committed up to 3
0925 03:04:59.450870 (+ 30us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:59.450921 (+ 51us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:59.451020 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:59.456132 24544 raft_consensus.cc:2396] T 8e5819c15eac4544a57e5ee6c9d4f117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 for term 7 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260925 03:04:59.443879 24586 leader_election.cc:302] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:59.449609 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25102ba08e4e49388f6176badcd10a25" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:04:59.455911 24541 rpcz_store.cc:276] 0925 03:04:58.416707 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:59.454318 (+1037611us) service_pool.cc:227] Handling call
0925 03:04:59.454595 (+ 277us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:59.454769 (+ 174us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:59.454792 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:59.454827 (+ 35us) raft_consensus.cc:1631] Marking committed up to 4
0925 03:04:59.454858 (+ 31us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:59.454901 (+ 43us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:59.455000 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:59.456808 24548 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 1158) took 1038 ms. Trace:
I20260925 03:04:59.459177 25199 raft_consensus.cc:1226] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [term 9 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 8. Current term is 9. Ops: []
I20260925 03:04:59.452250 24470 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1221) took 1527 ms. Trace:
I20260925 03:04:59.460176 24548 rpcz_store.cc:276] 0925 03:04:58.418579 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:59.456070 (+1037491us) service_pool.cc:227] Handling call
0925 03:04:59.456353 (+ 283us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:59.456506 (+ 153us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:59.456530 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:59.456566 (+ 36us) raft_consensus.cc:1631] Marking committed up to 4
0925 03:04:59.456594 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:59.456642 (+ 48us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:59.456753 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:59.459877 24470 rpcz_store.cc:276] 0925 03:04:57.924487 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:58.822941 (+898454us) service_pool.cc:227] Handling call
0925 03:04:58.823236 (+ 295us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:04:58.823596 (+ 360us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:58.823625 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:04:58.824034 (+ 409us) log.cc:842] Serialized 26 byte log entry
0925 03:04:59.156050 (+332016us) raft_consensus.cc:1631] Marking committed up to 3
0925 03:04:59.156089 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:59.156155 (+ 66us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:04:59.452058 (+295903us) raft_consensus.cc:1671] finished
0925 03:04:59.452080 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:59.452194 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":106,"raft.run_cpu_time_us":2576,"raft.run_wall_time_us":34930,"thread_start_us":219895,"threads_started":1,"wal-append.queue_time_us":603633}
I20260925 03:04:59.447386 24545 rpcz_store.cc:276] 0925 03:04:58.399855 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:59.416735 (+1016880us) service_pool.cc:227] Handling call
0925 03:04:59.417113 (+ 378us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:59.417345 (+ 232us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:59.417370 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:59.417413 (+ 43us) raft_consensus.cc:1631] Marking committed up to 2
0925 03:04:59.417444 (+ 31us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:59.417497 (+ 53us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:59.417605 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:59.456808 24539 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 1159) took 1006 ms. Trace:
I20260925 03:04:59.465077 24545 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a4abe0ea757e4886be4945c5001e529a"
dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
from {username='slave'} at 127.0.0.1:36786
I20260925 03:04:59.465665 24545 raft_consensus.cc:482] T a4abe0ea757e4886be4945c5001e529a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 03:04:59.460670 24541 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 1160) took 1006 ms. Trace:
W20260925 03:04:59.432337 26989 log.cc:924] Time spent T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43: Append to log took a long time: real 0.192s user 0.000s sys 0.000s
I20260925 03:04:59.464895 24539 rpcz_store.cc:276] 0925 03:04:58.450286 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:59.456120 (+1005834us) service_pool.cc:227] Handling call
0925 03:04:59.456384 (+ 264us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:59.456530 (+ 146us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:59.456551 (+ 21us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:59.456586 (+ 35us) raft_consensus.cc:1631] Marking committed up to 4
0925 03:04:59.456613 (+ 27us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:59.456653 (+ 40us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:59.456753 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":33024}
W20260925 03:04:59.502173 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 50565us
I20260925 03:04:59.503021 24437 leader_election.cc:302] T 11bb46bce3564841b7674e6a693bd2e6 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
W20260925 03:04:59.491039 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 44948us
I20260925 03:04:59.494403 24541 rpcz_store.cc:276] 0925 03:04:58.453983 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:59.459913 (+1005930us) service_pool.cc:227] Handling call
0925 03:04:59.460226 (+ 313us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:59.460399 (+ 173us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:59.460420 (+ 21us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:59.460456 (+ 36us) raft_consensus.cc:1631] Marking committed up to 3
0925 03:04:59.460487 (+ 31us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:59.460531 (+ 44us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:59.460617 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260925 03:04:59.505849 24514 leader_election.cc:334] T 8e5819c15eac4544a57e5ee6c9d4f117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187): Timed out: RequestConsensusVote RPC to 127.22.193.1:42187 timed out after 1.985s (SENT)
I20260925 03:04:59.506224 24514 leader_election.cc:302] T 8e5819c15eac4544a57e5ee6c9d4f117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:59.504346 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15861cd28e354598a3c6bfc4337c34bb" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:04:59.506955 24541 raft_consensus.cc:1261] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 5 index: 8. (index mismatch)
I20260925 03:04:59.507323 24466 raft_consensus.cc:2454] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 5.
I20260925 03:04:59.495267 24545 raft_consensus.cc:3037] T a4abe0ea757e4886be4945c5001e529a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Advancing to term 3
W20260925 03:04:59.344761 26846 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.682s user 0.008s sys 0.005s
I20260925 03:04:59.467161 24547 raft_consensus.cc:2379] T 25102ba08e4e49388f6176badcd10a25 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 in current term 5: Already voted for candidate eeab93abfdb141a895002ec9f312296c in this term.
W20260925 03:04:59.509723 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 229495us
W20260925 03:04:59.513630 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 69818us
I20260925 03:04:59.516882 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "100c113a9fd84cc8adf2ec7b33ea0093" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:59.517145 25199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6b6ec010caf4689b8c90449764f5cf9" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43"
I20260925 03:04:59.518532 24619 raft_consensus.cc:1261] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Refusing update from remote peer 1bc1f2e62ab44af995165bd5116a9d43: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260925 03:04:59.523413 24547 raft_consensus.cc:1261] T 550118bc1a0641ce9803bdbf43f2069f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260925 03:04:59.531111 24437 leader_election.cc:302] T 43366a0a2f9d4344bf4e37ffd290ef8a P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
W20260925 03:04:59.531659 26996 log.cc:924] Time spent T 367277daf8d04778b3fa4ed94d71580d P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.200s user 0.000s sys 0.000s
W20260925 03:04:59.513036 26846 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.851s user 0.009s sys 0.005s
W20260925 03:04:59.556077 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 50256.4us
W20260925 03:04:59.594650 24514 leader_election.cc:334] T 48ca6ccd98194cf2af7b5278d15bf4ea P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187): Timed out: RequestConsensusVote RPC to 127.22.193.1:42187 timed out after 2.004s (SENT)
I20260925 03:04:59.576458 24437 leader_election.cc:302] T ceab6b0fccd54e909a64f66fe81a9a1f P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:04:59.508998 26738 raft_consensus.cc:482] T 3d89396576884f7c8901dba54f783947 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:59.599798 24468 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1230) took 1673 ms. Trace:
I20260925 03:04:59.593693 26992 raft_consensus.cc:2726] T 8e5819c15eac4544a57e5ee6c9d4f117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260925 03:04:59.611135 24468 rpcz_store.cc:276] 0925 03:04:57.926839 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:58.862659 (+935820us) service_pool.cc:227] Handling call
0925 03:04:58.967804 (+105145us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:04:58.999332 (+ 31528us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:58.999362 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:04:58.999800 (+ 438us) log.cc:842] Serialized 26 byte log entry
0925 03:04:59.263801 (+264001us) raft_consensus.cc:1631] Marking committed up to 3
0925 03:04:59.599008 (+335207us) log.cc:842] Serialized 14 byte log entry
0925 03:04:59.599432 (+ 424us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:59.599517 (+ 85us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:04:59.599565 (+ 48us) raft_consensus.cc:1671] finished
0925 03:04:59.599576 (+ 11us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:59.599728 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":33169,"raft.queue_time_us":42917,"raft.run_cpu_time_us":164,"raft.run_wall_time_us":6884,"thread_start_us":380987,"threads_started":2,"wal-append.queue_time_us":418701}
I20260925 03:04:59.528313 26824 raft_consensus.cc:2726] T 11bb46bce3564841b7674e6a693bd2e6 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260925 03:04:59.529024 25191 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48424 (request call id 237) took 1160 ms. Trace:
I20260925 03:04:59.532136 25193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e068a0f80941dc9138cee063f53bab" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:59.580060 26883 raft_consensus.cc:2781] T 43366a0a2f9d4344bf4e37ffd290ef8a P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260925 03:04:59.466486 26804 raft_consensus.cc:482] T 71f461a6e2ff434993b78a2d077f0abd P 1bc1f2e62ab44af995165bd5116a9d43 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:59.595088 25199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15861cd28e354598a3c6bfc4337c34bb" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:59.595089 24514 leader_election.cc:302] T 48ca6ccd98194cf2af7b5278d15bf4ea P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:59.576260 24545 raft_consensus.cc:504] T a4abe0ea757e4886be4945c5001e529a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:59.649668 25199 raft_consensus.cc:2454] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 5.
W20260925 03:04:59.650142 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 55538.1us
W20260925 03:04:59.650826 24514 leader_election.cc:334] T 550118bc1a0641ce9803bdbf43f2069f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187): Timed out: RequestConsensusVote RPC to 127.22.193.1:42187 timed out after 2.009s (SENT)
W20260925 03:04:59.651346 26937 consensus_meta.cc:287] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent flushing consensus metadata: real 0.525s user 0.008s sys 0.001s
W20260925 03:04:59.651621 24514 leader_election.cc:334] T c85f5d1d3f004cf5817bb594fd8e6469 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187): Timed out: RequestConsensusVote RPC to 127.22.193.1:42187 timed out after 2.012s (SENT)
I20260925 03:04:59.650946 24545 leader_election.cc:288] T a4abe0ea757e4886be4945c5001e529a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 election: Requested vote from peers eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:59.597425 24437 leader_election.cc:302] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
W20260925 03:04:59.600744 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 79558us
W20260925 03:04:59.601300 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 79780.5us
I20260925 03:04:59.515529 26861 raft_consensus.cc:2726] T b627ecb4a5384ef392aa695e06599c39 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260925 03:04:59.611428 24543 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36786 (request call id 1156) took 1193 ms. Trace:
I20260925 03:04:59.658254 24543 rpcz_store.cc:276] 0925 03:04:58.417536 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:58.993940 (+576404us) service_pool.cc:227] Handling call
0925 03:04:59.611229 (+617289us) spinlock_profiling.cc:243] Waited 225 ms on lock 0x7b54003b6d54. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d42ae 00007f437b411c27 00007f4374af9c04 00007f4374af9b92 00007f4374af9ac1 00007f4374af9a3c 00007f4374af8cf2 00007f43748fa94c 00007f43748f9d86 00007f43748f972f
0925 03:04:59.611365 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":516690304}
I20260925 03:04:59.523857 26378 raft_consensus.cc:2781] T 442d4a29b1d743599f4b6b4871e26668 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260925 03:04:59.520532 26509 leader_election.cc:288] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:59.528672 27008 raft_consensus.cc:2781] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Leader election won for term 7
I20260925 03:04:59.506793 26909 raft_consensus.cc:504] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:59.618110 26878 raft_consensus.cc:2781] T ceab6b0fccd54e909a64f66fe81a9a1f P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260925 03:04:59.646128 24468 raft_consensus.cc:1261] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260925 03:04:59.612218 26985 raft_consensus.cc:2726] T 21c43af5e6844fdaacc8c625fa01fd8d P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260925 03:04:59.647971 25163 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 186418us
I20260925 03:04:59.652077 24514 leader_election.cc:302] T c85f5d1d3f004cf5817bb594fd8e6469 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:59.654031 24586 leader_election.cc:302] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5
W20260925 03:04:59.655938 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 58607.3us
I20260925 03:04:59.622597 26738 raft_consensus.cc:504] T 3d89396576884f7c8901dba54f783947 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:59.626135 26947 leader_election.cc:288] T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:59.661725 24546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 1141) took 1275 ms. Trace:
I20260925 03:04:59.642587 26883 raft_consensus.cc:482] T 43366a0a2f9d4344bf4e37ffd290ef8a P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:59.640564 26843 leader_election.cc:288] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:59.662894 26378 raft_consensus.cc:482] T 442d4a29b1d743599f4b6b4871e26668 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:04:59.683753 24437 leader_election.cc:302] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:59.662256 24545 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RunLeaderElection from 127.0.0.1:36786 (request call id 1165) took 1073 ms. Trace:
I20260925 03:04:59.645210 25193 raft_consensus.cc:2454] T 04e068a0f80941dc9138cee063f53bab P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 4.
I20260925 03:04:59.686941 25199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:04:59.661254 26955 raft_consensus.cc:2726] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260925 03:04:59.658166 26992 raft_consensus.cc:2726] T 48ca6ccd98194cf2af7b5278d15bf4ea P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260925 03:04:59.683081 24514 leader_election.cc:334] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187): Timed out: RequestConsensusVote RPC to 127.22.193.1:42187 timed out after 2.044s (SENT)
I20260925 03:04:59.690286 24546 rpcz_store.cc:276] 0925 03:04:58.385860 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:58.385987 (+ 127us) service_pool.cc:227] Handling call
0925 03:04:58.386273 (+ 286us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:04:58.399012 (+ 12739us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:58.399047 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:04:58.412359 (+ 13312us) log.cc:842] Serialized 26 byte log entry
0925 03:04:58.728975 (+316616us) raft_consensus.cc:1631] Marking committed up to 5
0925 03:04:58.729011 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:58.729080 (+ 69us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:04:59.661512 (+932432us) raft_consensus.cc:1671] finished
0925 03:04:59.661531 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:59.661666 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":10283,"raft.run_cpu_time_us":170,"raft.run_wall_time_us":8343,"thread_start_us":259730,"threads_started":1,"wal-append.queue_time_us":724885}
I20260925 03:04:59.710091 24514 leader_election.cc:302] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:59.678875 26878 raft_consensus.cc:482] T ceab6b0fccd54e909a64f66fe81a9a1f P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:59.703186 24545 rpcz_store.cc:276] 0925 03:04:58.578961 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:59.464970 (+886009us) service_pool.cc:227] Handling call
0925 03:04:59.652264 (+187294us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:04:59.648806 26923 consensus_queue.cc:1035] T 8ac466793d094325ace224708b2158ed P 1bc1f2e62ab44af995165bd5116a9d43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.036s
I20260925 03:04:59.701174 26883 raft_consensus.cc:3037] T 43366a0a2f9d4344bf4e37ffd290ef8a P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:04:59.706414 25199 raft_consensus.cc:2454] T 7e49248643df4131a1c42e74244e05c2 P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 6.
I20260925 03:04:59.651732 26937 raft_consensus.cc:504] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:59.709378 24437 leader_election.cc:302] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
W20260925 03:04:59.715651 24514 leader_election.cc:334] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187): Timed out: RequestConsensusVote RPC to 127.22.193.1:42187 timed out after 2.139s (SENT)
I20260925 03:04:59.688891 26811 raft_consensus.cc:2726] T c85f5d1d3f004cf5817bb594fd8e6469 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260925 03:04:59.660547 26804 raft_consensus.cc:504] T 71f461a6e2ff434993b78a2d077f0abd P 1bc1f2e62ab44af995165bd5116a9d43 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
W20260925 03:04:59.748167 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 93537.8us
I20260925 03:04:59.752164 24437 leader_election.cc:302] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
W20260925 03:04:59.753050 24514 leader_election.cc:334] T 7993f9c3e2a44f408bbf5d1e395e2306 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187): Timed out: RequestConsensusVote RPC to 127.22.193.1:42187 timed out after 2.446s (SENT)
I20260925 03:04:59.756675 24619 raft_consensus.cc:1226] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 8. Current term is 9. Ops: [8.9-8.9]
I20260925 03:04:59.707374 26378 raft_consensus.cc:3037] T 442d4a29b1d743599f4b6b4871e26668 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Advancing to term 7
W20260925 03:04:59.761446 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 107486us
I20260925 03:04:59.762192 24586 leader_election.cc:302] T 25102ba08e4e49388f6176badcd10a25 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:04:59.653684 26913 consensus_queue.cc:260] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:04:59.714265 26878 raft_consensus.cc:3037] T ceab6b0fccd54e909a64f66fe81a9a1f P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:04:59.758308 25199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4abe0ea757e4886be4945c5001e529a" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43"
I20260925 03:04:59.798286 25199 raft_consensus.cc:3037] T a4abe0ea757e4886be4945c5001e529a P 1bc1f2e62ab44af995165bd5116a9d43 [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:04:59.714000 26843 raft_consensus.cc:2726] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260925 03:04:59.747165 26955 raft_consensus.cc:2726] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260925 03:04:59.715466 26897 leader_election.cc:288] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 8 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:59.800630 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48ca6ccd98194cf2af7b5278d15bf4ea" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:59.718326 26979 raft_consensus.cc:2726] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260925 03:04:59.803418 26913 raft_consensus.cc:3037] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Advancing to term 9
W20260925 03:04:59.760396 27010 log.cc:924] Time spent T edcf6e6a422d4d198d4fbb0f7d3d845d P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.360s user 0.000s sys 0.001s
I20260925 03:04:59.747535 26956 leader_election.cc:288] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:59.796437 24540 raft_consensus.cc:3037] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:04:59.809863 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "420163a6cdce412ebc21c25fe3835638" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:59.761332 26800 leader_election.cc:288] T 550118bc1a0641ce9803bdbf43f2069f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:59.834455 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:04:59.791193 27008 raft_consensus.cc:686] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c [term 7 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:59.791005 26682 raft_consensus.cc:2781] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260925 03:04:59.798370 24514 leader_election.cc:302] T 7993f9c3e2a44f408bbf5d1e395e2306 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
W20260925 03:04:59.808377 27011 log.cc:924] Time spent T f22ddd7d1c444dc58f45a1fb0b3e02bc P 869b0a8575b54d08ae2817fad5dd42d5: Append to log took a long time: real 0.366s user 0.000s sys 0.001s
W20260925 03:04:59.840867 26984 consensus_meta.cc:287] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5: Time spent flushing consensus metadata: real 0.702s user 0.011s sys 0.000s
I20260925 03:04:59.758016 24437 leader_election.cc:302] T 0be4e0aa984f4f2b81a58e382e434ad5 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
W20260925 03:04:59.876405 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 123393us
I20260925 03:04:59.878798 24471 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1220) took 1954 ms. Trace:
I20260925 03:04:59.835141 26756 raft_consensus.cc:2781] T 25102ba08e4e49388f6176badcd10a25 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Leader election won for term 5
W20260925 03:04:59.879604 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 121648us
I20260925 03:04:59.879189 24471 rpcz_store.cc:276] 0925 03:04:57.924219 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:58.792483 (+868264us) service_pool.cc:227] Handling call
0925 03:04:59.444850 (+652367us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:04:59.455733 (+ 10883us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:59.878593 (+422860us) spinlock_profiling.cc:243] Waited 652 ms on lock 0x7b54001919d0. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764cfa09 00007f437b4112e9 00007f4374af7684 00007f4374af7612 00007f4374af7541 00007f4374af74bc 00007f4374af6772 00007f43748fa94c 00007f43748f9d86 00007f43748f972f
0925 03:04:59.878725 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1499549568}
I20260925 03:04:59.802186 27009 raft_consensus.cc:2781] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Leader election won for term 6
I20260925 03:04:59.886873 25199 raft_consensus.cc:2454] T a4abe0ea757e4886be4945c5001e529a P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 3.
W20260925 03:04:59.884339 24514 connection.cc:232] RPC call timeout handler was delayed by -1.1397836960004497s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19190000000000002s for a response
I20260925 03:04:59.875461 26956 raft_consensus.cc:482] T 11bb46bce3564841b7674e6a693bd2e6 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:59.887621 24437 leader_election.cc:302] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:04:59.889307 24511 leader_election.cc:302] T a4abe0ea757e4886be4945c5001e529a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:04:59.898990 24437 leader_election.cc:302] T 8e5819c15eac4544a57e5ee6c9d4f117 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:59.884756 24539 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 1169) took 1260 ms. Trace:
I20260925 03:04:59.909525 24437 leader_election.cc:302] T 25102ba08e4e49388f6176badcd10a25 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:59.897030 24540 raft_consensus.cc:2454] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 7.
I20260925 03:04:59.867086 26682 raft_consensus.cc:482] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:04:59.876179 26979 raft_consensus.cc:2726] T 7993f9c3e2a44f408bbf5d1e395e2306 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260925 03:04:59.909785 26897 raft_consensus.cc:3037] T 8e5819c15eac4544a57e5ee6c9d4f117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:04:59.857214 26712 consensus_queue.cc:1035] T 1414dcfe53f64a78ae413531eb8ec117 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.051s
I20260925 03:04:59.876945 26909 leader_election.cc:288] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:04:59.878218 27022 raft_consensus.cc:482] T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:04:59.890111 26992 raft_consensus.cc:482] T 8ac466793d094325ace224708b2158ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:04:59.629175 25191 rpcz_store.cc:276] 0925 03:04:58.363835 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:58.368295 (+ 4460us) service_pool.cc:227] Handling call
0925 03:04:58.397245 (+ 28950us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:04:58.457046 (+ 59801us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:58.461870 (+ 4824us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:04:58.577749 (+115879us) log.cc:842] Serialized 26 byte log entry
0925 03:04:59.050836 (+473087us) raft_consensus.cc:1631] Marking committed up to 7
0925 03:04:59.054620 (+ 3784us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:59.060414 (+ 5794us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:04:59.497926 (+437512us) raft_consensus.cc:1671] finished
0925 03:04:59.497949 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:59.524105 (+ 26156us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":9347,"raft.run_cpu_time_us":153,"raft.run_wall_time_us":16235,"thread_start_us":407395,"threads_started":1,"wal-append.queue_time_us":624247}
I20260925 03:04:59.879778 26984 raft_consensus.cc:504] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:04:59.952379 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 118317us
I20260925 03:04:59.956264 24472 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1217) took 2033 ms. Trace:
I20260925 03:04:59.888216 26956 raft_consensus.cc:504] T 11bb46bce3564841b7674e6a693bd2e6 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:04:59.957234 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 176696us
I20260925 03:04:59.956744 24472 rpcz_store.cc:276] 0925 03:04:57.923377 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:58.789821 (+866444us) service_pool.cc:227] Handling call
0925 03:04:58.790139 (+ 318us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:04:58.790491 (+ 352us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:58.790520 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:04:58.791003 (+ 483us) log.cc:842] Serialized 26 byte log entry
0925 03:04:59.043306 (+252303us) raft_consensus.cc:1631] Marking committed up to 3
0925 03:04:59.043344 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:59.043429 (+ 85us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:04:59.953700 (+910271us) raft_consensus.cc:1671] finished
0925 03:04:59.953719 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:59.956193 (+ 2474us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":813,"raft.run_cpu_time_us":79,"raft.run_wall_time_us":79,"spinlock_wait_cycles":3968,"thread_start_us":251923,"threads_started":1,"wal-append.queue_time_us":437520}
I20260925 03:04:59.959616 24469 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1252) took 1382 ms. Trace:
I20260925 03:04:59.911756 26882 raft_consensus.cc:2781] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Leader election won for term 8
I20260925 03:04:59.915688 26861 raft_consensus.cc:3037] T 25102ba08e4e49388f6176badcd10a25 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:04:59.915303 26800 raft_consensus.cc:2781] T a4abe0ea757e4886be4945c5001e529a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 03:04:59.915556 26682 raft_consensus.cc:3037] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:04:59.956516 24514 leader_election.cc:302] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:04:59.908860 26955 raft_consensus.cc:2781] T 0be4e0aa984f4f2b81a58e382e434ad5 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260925 03:04:59.966326 24542 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 1143) took 1575 ms. Trace:
I20260925 03:04:59.967329 24514 leader_election.cc:302] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:04:59.912899 27009 raft_consensus.cc:686] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P eeab93abfdb141a895002ec9f312296c [term 6 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
W20260925 03:04:59.949564 27012 log.cc:924] Time spent T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.322s user 0.000s sys 0.001s
I20260925 03:04:59.948849 26738 leader_election.cc:288] T 3d89396576884f7c8901dba54f783947 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:04:59.967841 24542 rpcz_store.cc:276] 0925 03:04:58.391068 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:58.393117 (+ 2049us) service_pool.cc:227] Handling call
0925 03:04:58.415741 (+ 22624us) raft_consensus.cc:1457] Updating replica for 2 ops
0925 03:04:58.715171 (+299430us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:58.715200 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0925 03:04:58.740868 (+ 25668us) log.cc:842] Serialized 52 byte log entry
0925 03:04:59.030521 (+289653us) raft_consensus.cc:1631] Marking committed up to 1
0925 03:04:59.030576 (+ 55us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:59.030716 (+ 140us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:04:59.966093 (+935377us) raft_consensus.cc:1671] finished
0925 03:04:59.966113 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:59.966251 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":14549,"raft.queue_time_us":45463,"raft.run_cpu_time_us":114,"raft.run_wall_time_us":2914,"thread_start_us":532640,"threads_started":2,"wal-append.queue_time_us":427508}
I20260925 03:04:59.914297 26756 raft_consensus.cc:686] T 25102ba08e4e49388f6176badcd10a25 P eeab93abfdb141a895002ec9f312296c [term 5 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:04:59.915306 26843 consensus_queue.cc:1035] T 293ef69caf514ac98510a68495f54eda P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.051s
I20260925 03:04:59.922036 26947 consensus_queue.cc:1035] T 550118bc1a0641ce9803bdbf43f2069f P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.048s
I20260925 03:04:59.960052 24469 rpcz_store.cc:276] 0925 03:04:58.577531 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:59.447024 (+869493us) service_pool.cc:227] Handling call
0925 03:04:59.498929 (+ 51905us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:59.499156 (+ 227us) raft_consensus.cc:1520] Early marking committed up to index 3
0925 03:04:59.627459 (+128303us) log.cc:842] Serialized 14 byte log entry
0925 03:04:59.880680 (+253221us) log.cc:842] Serialized 14 byte log entry
0925 03:04:59.959179 (+ 78499us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:59.959241 (+ 62us) raft_consensus.cc:1631] Marking committed up to 3
0925 03:04:59.959352 (+ 111us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:59.959433 (+ 81us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:59.959553 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":90364,"prepare.queue_time_us":219030,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":40,"raft.queue_time_us":85228,"raft.run_wall_time_us":23845,"spinlock_wait_cycles":17651328,"thread_start_us":354685,"threads_started":2}
I20260925 03:04:59.975889 24473 raft_consensus.cc:1226] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 4. Current term is 5. Ops: []
I20260925 03:04:59.919585 26824 raft_consensus.cc:482] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260925 03:04:59.969949 25165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 394437us
I20260925 03:04:59.997844 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71f461a6e2ff434993b78a2d077f0abd" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:04:59.960793 27021 raft_consensus.cc:482] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:00.029698 25165 leader_election.cc:302] T 29637f5a4a504ce6ba29033ccd0cd716 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
W20260925 03:05:00.030478 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 67146.9us
W20260925 03:04:59.988407 27016 log.cc:924] Time spent T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.310s user 0.001s sys 0.000s
I20260925 03:05:00.041574 25165 leader_election.cc:302] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5; no voters:
W20260925 03:05:00.047510 27042 log.cc:924] Time spent T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.181s user 0.000s sys 0.001s
I20260925 03:05:00.048516 26979 raft_consensus.cc:2726] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260925 03:05:00.049263 26979 raft_consensus.cc:2726] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260925 03:05:00.053267 27043 log.cc:924] Time spent T 1414dcfe53f64a78ae413531eb8ec117 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.088s user 0.000s sys 0.000s
I20260925 03:05:00.052994 26883 raft_consensus.cc:504] T 43366a0a2f9d4344bf4e37ffd290ef8a P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:05:00.054759 27028 log.cc:924] Time spent T cc2df40db08649c9a75e90000b878559 P 869b0a8575b54d08ae2817fad5dd42d5: Append to log took a long time: real 0.107s user 0.000s sys 0.000s
I20260925 03:05:00.055138 26883 leader_election.cc:288] T 43366a0a2f9d4344bf4e37ffd290ef8a P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:59.979605 26937 leader_election.cc:288] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 6 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:00.055866 24467 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1232) took 2128 ms. Trace:
I20260925 03:05:00.056391 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43366a0a2f9d4344bf4e37ffd290ef8a" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:05:00.056962 24544 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 1166) took 1455 ms. Trace:
I20260925 03:04:59.977787 26804 leader_election.cc:288] T 71f461a6e2ff434993b78a2d077f0abd P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:04:59.970175 26955 raft_consensus.cc:482] T 0be4e0aa984f4f2b81a58e382e434ad5 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:00.057755 26955 raft_consensus.cc:3037] T 0be4e0aa984f4f2b81a58e382e434ad5 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:05:00.056241 24467 rpcz_store.cc:276] 0925 03:04:57.927350 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:58.939406 (+1012056us) service_pool.cc:227] Handling call
0925 03:04:58.939783 (+ 377us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:04:59.013210 (+ 73427us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:59.013248 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:04:59.021000 (+ 7752us) log.cc:842] Serialized 26 byte log entry
0925 03:04:59.345157 (+324157us) raft_consensus.cc:1631] Marking committed up to 2
0925 03:04:59.345202 (+ 45us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:59.345282 (+ 80us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:05:00.055673 (+710391us) raft_consensus.cc:1671] finished
0925 03:05:00.055691 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:05:00.055813 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":31847,"raft.queue_time_us":48151,"raft.run_cpu_time_us":136,"raft.run_wall_time_us":11248,"thread_start_us":270273,"threads_started":1,"wal-append.queue_time_us":862422}
I20260925 03:05:00.063473 26897 raft_consensus.cc:2726] T 8e5819c15eac4544a57e5ee6c9d4f117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260925 03:05:00.057319 24544 rpcz_store.cc:276] 0925 03:04:58.601333 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:59.493953 (+892620us) service_pool.cc:227] Handling call
0925 03:04:59.522825 (+ 28872us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:04:59.529955 (+ 7130us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:59.529984 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:04:59.600359 (+ 70375us) log.cc:842] Serialized 26 byte log entry
0925 03:04:59.747116 (+146757us) raft_consensus.cc:1631] Marking committed up to 2
0925 03:04:59.747171 (+ 55us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:59.747246 (+ 75us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:05:00.056758 (+309512us) raft_consensus.cc:1671] finished
0925 03:05:00.056779 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:05:00.056905 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6750,"raft.queue_time_us":8579,"raft.run_cpu_time_us":54,"raft.run_wall_time_us":54,"thread_start_us":110664,"threads_started":1,"wal-append.queue_time_us":286872}
I20260925 03:05:00.066037 26378 raft_consensus.cc:504] T 442d4a29b1d743599f4b6b4871e26668 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:00.068137 26378 leader_election.cc:288] T 442d4a29b1d743599f4b6b4871e26668 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 7 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:05:00.070797 26861 raft_consensus.cc:2726] T 25102ba08e4e49388f6176badcd10a25 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260925 03:05:00.072614 27017 log.cc:924] Time spent T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5: Append to log took a long time: real 0.553s user 0.000s sys 0.001s
I20260925 03:05:00.074250 26984 leader_election.cc:288] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:00.074167 24465 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1229) took 2147 ms. Trace:
I20260925 03:05:00.074580 26955 raft_consensus.cc:504] T 0be4e0aa984f4f2b81a58e382e434ad5 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:59.964159 26992 raft_consensus.cc:504] T 8ac466793d094325ace224708b2158ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:00.074744 24465 rpcz_store.cc:276] 0925 03:04:57.926609 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:58.858519 (+931910us) service_pool.cc:227] Handling call
0925 03:04:58.858820 (+ 301us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:04:58.859198 (+ 378us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:58.859224 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:04:58.859692 (+ 468us) log.cc:842] Serialized 26 byte log entry
0925 03:04:59.191080 (+331388us) raft_consensus.cc:1631] Marking committed up to 5
0925 03:04:59.191121 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:59.191195 (+ 74us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:05:00.072007 (+880812us) spinlock_profiling.cc:243] Waited 3.89 ms on lock 0x7b44000232cc. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f43733ecf11 00007f43734baa95 00007f43734bb4b2 00007f4376163511 00007f437615ee1a 00007f4376152bee 00007f4376151f36 00007f4376151b58 00007f437615b551 00007f437615b509 00007f437615b499
0925 03:05:00.073868 (+ 1861us) raft_consensus.cc:1671] finished
0925 03:05:00.073889 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:05:00.074109 (+ 220us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4629,"raft.queue_time_us":5627,"raft.run_cpu_time_us":152,"raft.run_wall_time_us":43127,"spinlock_wait_cycles":8949120,"thread_start_us":215405,"threads_started":1,"wal-append.queue_time_us":400851}
I20260925 03:05:00.075965 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5278a60f97da4e10aecd50a8d9a9a43c" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:05:00.076603 26955 leader_election.cc:288] T 0be4e0aa984f4f2b81a58e382e434ad5 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:00.076867 26992 leader_election.cc:288] T 8ac466793d094325ace224708b2158ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:05:00.081961 25199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c81a1b3b616492ab6c964f45af36077" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
W20260925 03:05:00.029263 27023 log.cc:924] Time spent T c88cbd39e01f44628a094af516be2833 P 869b0a8575b54d08ae2817fad5dd42d5: Append to log took a long time: real 0.333s user 0.000s sys 0.001s
I20260925 03:05:00.078568 24541 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 1503) took 1307 ms. Trace:
W20260925 03:05:00.084491 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 119623us
I20260925 03:04:59.988740 26882 raft_consensus.cc:686] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 LEADER]: Becoming Leader. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:05:00.098258 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15861cd28e354598a3c6bfc4337c34bb" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:00.101361 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "100c113a9fd84cc8adf2ec7b33ea0093" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
W20260925 03:05:00.104770 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 73621us
I20260925 03:05:00.085467 24541 rpcz_store.cc:276] 0925 03:04:58.771622 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:59.511565 (+739943us) service_pool.cc:227] Handling call
0925 03:04:59.511884 (+ 319us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:04:59.529545 (+ 17661us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:59.529576 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:04:59.530015 (+ 439us) log.cc:842] Serialized 26 byte log entry
0925 03:04:59.716103 (+186088us) raft_consensus.cc:1631] Marking committed up to 4
0925 03:04:59.957221 (+241118us) log.cc:842] Serialized 14 byte log entry
0925 03:04:59.957540 (+ 319us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:59.957635 (+ 95us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:05:00.078347 (+120712us) raft_consensus.cc:1671] finished
0925 03:05:00.078369 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:05:00.078505 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9964,"prepare.queue_time_us":229639,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":42,"raft.queue_time_us":19362,"raft.run_cpu_time_us":109,"raft.run_wall_time_us":88,"spinlock_wait_cycles":6247296,"thread_start_us":289227,"threads_started":2,"wal-append.queue_time_us":240970}
I20260925 03:05:00.138022 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "420163a6cdce412ebc21c25fe3835638" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:00.139266 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:05:00.139807 24541 raft_consensus.cc:3037] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Advancing to term 8
W20260925 03:05:00.049472 27027 log.cc:924] Time spent T a7aa47c48404465b94a36fd819fe3c7a P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.165s user 0.001s sys 0.000s
W20260925 03:05:00.141006 25196 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.516s user 0.007s sys 0.030s
I20260925 03:05:00.143810 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48ca6ccd98194cf2af7b5278d15bf4ea" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:00.146502 24547 raft_consensus.cc:2454] T 48ca6ccd98194cf2af7b5278d15bf4ea P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 4.
I20260925 03:05:00.145046 24436 leader_election.cc:302] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5; no voters:
W20260925 03:05:00.146914 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 179673us
W20260925 03:05:00.147848 25196 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.523s user 0.008s sys 0.030s
I20260925 03:04:59.965272 27022 raft_consensus.cc:504] T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:04:59.964561 27008 consensus_queue.cc:237] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:00.181034 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5278a60f97da4e10aecd50a8d9a9a43c" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
W20260925 03:05:00.084681 26970 consensus_meta.cc:287] T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c: Time spent flushing consensus metadata: real 1.063s user 0.008s sys 0.004s
I20260925 03:05:00.143234 24543 raft_consensus.cc:2454] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 4.
I20260925 03:05:00.153450 24437 leader_election.cc:302] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:00.105774 26992 raft_consensus.cc:465] T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 LEADER]: Not starting pre-election -- already a leader
I20260925 03:05:00.150576 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43366a0a2f9d4344bf4e37ffd290ef8a" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:05:00.106284 27024 raft_consensus.cc:2781] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260925 03:05:00.149072 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6b6ec010caf4689b8c90449764f5cf9" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:05:00.149072 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be4e0aa984f4f2b81a58e382e434ad5" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:05:00.172731 24544 raft_consensus.cc:1261] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 5. (term mismatch)
I20260925 03:05:00.175757 24470 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1254) took 1587 ms. Trace:
I20260925 03:05:00.170872 26874 raft_consensus.cc:2781] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260925 03:05:00.203821 24544 pending_rounds.cc:76] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Aborting all ops after (but not including) 4
I20260925 03:05:00.205495 24544 pending_rounds.cc:97] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Aborting uncommitted NO_OP operation due to leader change: 7.5
I20260925 03:05:00.205785 24544 raft_consensus.cc:2866] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260925 03:05:00.188280 27016 log.cc:924] Time spent T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.082s user 0.001s sys 0.000s
W20260925 03:05:00.208251 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 65266.9us
I20260925 03:05:00.204550 24470 rpcz_store.cc:276] 0925 03:04:58.588012 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:59.461787 (+873775us) service_pool.cc:227] Handling call
0925 03:04:59.462048 (+ 261us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:59.462213 (+ 165us) raft_consensus.cc:1520] Early marking committed up to index 4
0925 03:04:59.961133 (+498920us) log.cc:842] Serialized 14 byte log entry
0925 03:05:00.175457 (+214324us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:05:00.175506 (+ 49us) raft_consensus.cc:1631] Marking committed up to 4
0925 03:05:00.175548 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:05:00.175610 (+ 62us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:05:00.175712 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":249627,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":37,"raft.queue_time_us":102,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":6496,"thread_start_us":666931,"threads_started":2}
I20260925 03:05:00.183027 24541 raft_consensus.cc:2396] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 for term 8 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
W20260925 03:05:00.152366 24585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.4:43979, user_credentials={real_user=slave}} blocked reactor thread for 72416.5us
I20260925 03:05:00.184464 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c85f5d1d3f004cf5817bb594fd8e6469" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:00.187719 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be4e0aa984f4f2b81a58e382e434ad5" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:05:00.188889 24437 leader_election.cc:302] T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:05:00.028956 26824 raft_consensus.cc:504] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:00.148016 24514 leader_election.cc:302] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:00.204909 24545 raft_consensus.cc:1261] T c88cbd39e01f44628a094af516be2833 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260925 03:05:00.211037 24437 leader_election.cc:302] T 100c113a9fd84cc8adf2ec7b33ea0093 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:00.212610 24548 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 1163) took 1651 ms. Trace:
I20260925 03:05:00.213156 24548 rpcz_store.cc:276] 0925 03:04:58.561144 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:59.462410 (+901266us) service_pool.cc:227] Handling call
0925 03:04:59.462646 (+ 236us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:59.462808 (+ 162us) raft_consensus.cc:1520] Early marking committed up to index 4
0925 03:04:59.859573 (+396765us) log.cc:842] Serialized 14 byte log entry
0925 03:05:00.212285 (+352712us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:05:00.212334 (+ 49us) raft_consensus.cc:1631] Marking committed up to 4
0925 03:05:00.212380 (+ 46us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:05:00.212452 (+ 72us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:05:00.212558 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":390240,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":47,"thread_start_us":699869,"threads_started":2}
I20260925 03:05:00.213824 24437 leader_election.cc:302] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
W20260925 03:05:00.214751 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 66941.1us
I20260925 03:05:00.216614 24514 leader_election.cc:302] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:00.204097 26800 raft_consensus.cc:686] T a4abe0ea757e4886be4945c5001e529a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 LEADER]: Becoming Leader. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:05:00.215829 24437 leader_election.cc:302] T 43366a0a2f9d4344bf4e37ffd290ef8a P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:00.220880 24464 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1244) took 1870 ms. Trace:
I20260925 03:05:00.193023 26861 raft_consensus.cc:2726] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260925 03:05:00.221526 24437 leader_election.cc:302] T e6b6ec010caf4689b8c90449764f5cf9 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: 1bc1f2e62ab44af995165bd5116a9d43, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:05:00.221237 24464 rpcz_store.cc:276] 0925 03:04:58.350721 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:59.048400 (+697679us) service_pool.cc:227] Handling call
0925 03:04:59.103048 (+ 54648us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:59.201333 (+ 98285us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:59.201364 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:59.201403 (+ 39us) raft_consensus.cc:1631] Marking committed up to 4
0925 03:04:59.201436 (+ 33us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:05:00.220649 (+1019213us) spinlock_profiling.cc:243] Waited 98.1 ms on lock 0x7b54002f17d4. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d0f51 00007f43764cf99e 00007f437b4112e9 00007f4374af7684 00007f4374af7612 00007f4374af7541 00007f4374af74bc 00007f4374af6772 00007f43748fa94c 00007f43748f9d86
0925 03:05:00.220717 (+ 68us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:05:00.220826 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":225575680}
I20260925 03:05:00.217833 24514 leader_election.cc:302] T a7aa47c48404465b94a36fd819fe3c7a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:00.201882 27024 raft_consensus.cc:482] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:05:00.042236 27021 raft_consensus.cc:504] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:00.226583 24437 leader_election.cc:302] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:00.202585 25193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43"
I20260925 03:05:00.205606 26984 raft_consensus.cc:482] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1bc1f2e62ab44af995165bd5116a9d43)
I20260925 03:05:00.238368 26984 raft_consensus.cc:504] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:00.240835 27058 raft_consensus.cc:2726] T a7aa47c48404465b94a36fd819fe3c7a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260925 03:05:00.243633 25199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c81a1b3b616492ab6c964f45af36077" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:05:00.216079 26947 raft_consensus.cc:2781] T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260925 03:05:00.230079 24514 leader_election.cc:302] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:00.209334 26874 raft_consensus.cc:482] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260925 03:05:00.234344 27054 log.cc:924] Time spent T 71f461a6e2ff434993b78a2d077f0abd P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.060s user 0.000s sys 0.000s
I20260925 03:05:00.214713 25192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ac466793d094325ace224708b2158ed" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:05:00.218252 26897 raft_consensus.cc:2726] T 100c113a9fd84cc8adf2ec7b33ea0093 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260925 03:05:00.287899 24585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.4:43979, user_credentials={real_user=slave}} blocked reactor thread for 62325.8us
W20260925 03:05:00.305474 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 78838.2us
I20260925 03:05:00.226125 26937 raft_consensus.cc:3037] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:05:00.309127 24585 leader_election.cc:302] T 04e068a0f80941dc9138cee063f53bab P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:05:00.220984 26883 raft_consensus.cc:2726] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260925 03:05:00.363101 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c81a1b3b616492ab6c964f45af36077" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:00.228295 25191 raft_consensus.cc:1261] T e92f377a79354a6ebb4eec67303c13e3 P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260925 03:05:00.188169 26756 consensus_queue.cc:237] T 25102ba08e4e49388f6176badcd10a25 P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:00.191382 26970 raft_consensus.cc:504] T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
W20260925 03:05:00.313140 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 104185us
I20260925 03:05:00.314136 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e5819c15eac4544a57e5ee6c9d4f117" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:05:00.314838 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e068a0f80941dc9138cee063f53bab" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:00.313144 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9c33fd100ff44c1a3247319a324a044" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:05:00.372349 24541 raft_consensus.cc:3037] T 8e5819c15eac4544a57e5ee6c9d4f117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Advancing to term 7
W20260925 03:05:00.317267 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 90753us
W20260925 03:05:00.310989 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 81147.1us
I20260925 03:05:00.231518 27024 raft_consensus.cc:3037] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43 [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:05:00.288166 26947 raft_consensus.cc:482] T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:00.366168 24436 leader_election.cc:302] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: 1bc1f2e62ab44af995165bd5116a9d43, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:00.229102 26843 raft_consensus.cc:2726] T 43366a0a2f9d4344bf4e37ffd290ef8a P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260925 03:05:00.232590 26979 raft_consensus.cc:2726] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260925 03:05:00.312799 27058 raft_consensus.cc:2726] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260925 03:05:00.370939 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5f9b503b0ab43999d582015a367441e" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:05:00.231645 26956 leader_election.cc:288] T 11bb46bce3564841b7674e6a693bd2e6 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:00.388370 24546 raft_consensus.cc:2379] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c in current term 8: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:05:00.318282 26874 raft_consensus.cc:3037] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:05:00.373327 24437 leader_election.cc:302] T 0be4e0aa984f4f2b81a58e382e434ad5 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:00.391678 24437 leader_election.cc:302] T c85f5d1d3f004cf5817bb594fd8e6469 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:00.245553 26984 leader_election.cc:288] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:05:00.232928 27019 raft_consensus.cc:2726] T e6b6ec010caf4689b8c90449764f5cf9 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260925 03:05:00.370733 24542 raft_consensus.cc:1261] T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260925 03:05:00.379187 24514 leader_election.cc:302] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 1bc1f2e62ab44af995165bd5116a9d43
I20260925 03:05:00.174125 27009 consensus_queue.cc:237] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:00.379817 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e8af2641e124f9589c5ccfdac785f63" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:00.379300 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d89396576884f7c8901dba54f783947" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:00.382943 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442d4a29b1d743599f4b6b4871e26668" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:05:00.370842 26897 raft_consensus.cc:482] T 37a157996e944db2906110d87ecb0c54 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:00.386054 24541 raft_consensus.cc:2454] T 8e5819c15eac4544a57e5ee6c9d4f117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 7.
W20260925 03:05:00.376253 27046 log.cc:924] Time spent T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.340s user 0.000s sys 0.001s
W20260925 03:05:00.383808 27054 log.cc:924] Time spent T 71f461a6e2ff434993b78a2d077f0abd P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.065s user 0.000s sys 0.000s
I20260925 03:05:00.400760 24544 raft_consensus.cc:3037] T 442d4a29b1d743599f4b6b4871e26668 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Advancing to term 7
I20260925 03:05:00.287072 26861 consensus_queue.cc:1035] T 06f4a7eaf6534e4986c9fd35911038e4 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.095s
I20260925 03:05:00.379732 26947 raft_consensus.cc:3037] T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:05:00.406392 24586 leader_election.cc:302] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:05:00.417698 24546 raft_consensus.cc:1226] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 7. Current term is 8. Ops: []
I20260925 03:05:00.367627 26955 raft_consensus.cc:2726] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260925 03:05:00.400714 24548 raft_consensus.cc:2454] T 3d89396576884f7c8901dba54f783947 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 4.
I20260925 03:05:00.410604 24514 leader_election.cc:302] T 21c43af5e6844fdaacc8c625fa01fd8d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:05:00.441220 24514 leader_election.cc:302] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:00.396615 26843 raft_consensus.cc:2726] T 0be4e0aa984f4f2b81a58e382e434ad5 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260925 03:05:00.441557 24586 leader_election.cc:302] T 3d89396576884f7c8901dba54f783947 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:05:00.230193 26882 consensus_queue.cc:237] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:00.383065 26883 raft_consensus.cc:2726] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260925 03:05:00.387895 26980 consensus_queue.cc:1035] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.060s
I20260925 03:05:00.421015 26861 consensus_queue.cc:1225] T 06f4a7eaf6534e4986c9fd35911038e4 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Peer a04a37fc4aa7460cbfdbeeb4e451e1f5 log is divergent from this leader: its last log entry 5.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260925 03:05:00.460461 24514 leader_election.cc:302] T f22ddd7d1c444dc58f45a1fb0b3e02bc P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:00.465370 24514 leader_election.cc:398] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187) with higher term. Message: Invalid argument: T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 6. Current term is 7.
I20260925 03:05:00.466226 24514 leader_election.cc:302] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:05:00.469422 27074 raft_consensus.cc:2781] T 04e068a0f80941dc9138cee063f53bab P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260925 03:05:00.470252 27022 leader_election.cc:288] T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:05:00.471068 27074 raft_consensus.cc:482] T 04e068a0f80941dc9138cee063f53bab P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting leader election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:04:59.900502 24539 rpcz_store.cc:276] 0925 03:04:58.624419 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:59.499750 (+875331us) service_pool.cc:227] Handling call
0925 03:04:59.499992 (+ 242us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:59.500178 (+ 186us) raft_consensus.cc:1520] Early marking committed up to index 4
0925 03:04:59.672373 (+172195us) log.cc:842] Serialized 14 byte log entry
0925 03:04:59.884393 (+212020us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:59.884445 (+ 52us) raft_consensus.cc:1631] Marking committed up to 4
0925 03:04:59.884498 (+ 53us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:59.884562 (+ 64us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:04:59.884680 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11195,"raft.queue_time_us":953,"raft.run_cpu_time_us":133,"raft.run_wall_time_us":6280,"thread_start_us":288433,"threads_started":2}
I20260925 03:05:00.447454 26956 raft_consensus.cc:2781] T 21c43af5e6844fdaacc8c625fa01fd8d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260925 03:05:00.491698 24511 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.22.193.3:35105 timed out after 1.758s (SENT) [suppressed 20 similar messages]
W20260925 03:05:00.480253 24436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 83256.5us
W20260925 03:05:00.519716 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 141684us
I20260925 03:05:00.414762 26897 raft_consensus.cc:504] T 37a157996e944db2906110d87ecb0c54 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:00.528313 24544 raft_consensus.cc:2454] T 442d4a29b1d743599f4b6b4871e26668 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 7.
I20260925 03:05:00.472637 27074 raft_consensus.cc:3037] T 04e068a0f80941dc9138cee063f53bab P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Advancing to term 5
W20260925 03:05:00.536368 26878 consensus_meta.cc:287] T ceab6b0fccd54e909a64f66fe81a9a1f P 869b0a8575b54d08ae2817fad5dd42d5: Time spent flushing consensus metadata: real 0.725s user 0.006s sys 0.005s
I20260925 03:05:00.480285 26980 raft_consensus.cc:2781] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260925 03:05:00.543720 26754 consensus_meta.cc:287] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c: Time spent flushing consensus metadata: real 1.315s user 0.014s sys 0.013s
I20260925 03:05:00.538450 26956 raft_consensus.cc:482] T 21c43af5e6844fdaacc8c625fa01fd8d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:00.540581 27055 raft_consensus.cc:2781] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Leader election won for term 7
I20260925 03:05:00.543627 26824 leader_election.cc:288] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:00.581257 24620 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38570 (request call id 1568) took 1050 ms. Trace:
I20260925 03:05:00.451236 26800 consensus_queue.cc:237] T a4abe0ea757e4886be4945c5001e529a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:00.597783 27066 consensus_queue.cc:1035] T c88cbd39e01f44628a094af516be2833 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.189s
I20260925 03:05:00.599362 27067 raft_consensus.cc:2781] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260925 03:05:00.581892 24620 rpcz_store.cc:276] 0925 03:04:59.526852 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:59.526990 (+ 138us) service_pool.cc:227] Handling call
0925 03:04:59.527333 (+ 343us) raft_consensus.cc:1457] Updating replica for 2 ops
0925 03:04:59.661710 (+134377us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:59.661746 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0925 03:04:59.673305 (+ 11559us) log.cc:842] Serialized 52 byte log entry
0925 03:04:59.876772 (+203467us) raft_consensus.cc:1631] Marking committed up to 3
0925 03:04:59.876810 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:04:59.876888 (+ 78us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:05:00.577223 (+700335us) raft_consensus.cc:1671] finished
0925 03:05:00.577245 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:05:00.577393 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":73676,"raft.queue_time_us":156211,"raft.run_cpu_time_us":155,"raft.run_wall_time_us":12483,"thread_start_us":258065,"threads_started":2,"wal-append.queue_time_us":275147}
I20260925 03:05:00.561100 26980 raft_consensus.cc:482] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:05:00.578733 26956 raft_consensus.cc:3037] T 21c43af5e6844fdaacc8c625fa01fd8d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:05:00.578534 27022 raft_consensus.cc:2781] T 3d89396576884f7c8901dba54f783947 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260925 03:05:00.552121 27007 consensus_queue.cc:1035] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.164s
I20260925 03:05:00.599951 27067 raft_consensus.cc:482] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:05:00.585086 26843 raft_consensus.cc:482] T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:00.578984 26754 raft_consensus.cc:504] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:05:00.756240 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 232315us
I20260925 03:05:00.718578 26861 raft_consensus.cc:482] T 1414dcfe53f64a78ae413531eb8ec117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:00.600281 26984 raft_consensus.cc:482] T 3d89396576884f7c8901dba54f783947 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:00.572213 26878 raft_consensus.cc:504] T ceab6b0fccd54e909a64f66fe81a9a1f P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:00.606566 27055 raft_consensus.cc:686] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c [term 7 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:05:00.659808 26980 raft_consensus.cc:3037] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Advancing to term 8
I20260925 03:05:00.719556 27066 raft_consensus.cc:482] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260925 03:05:00.792680 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 326508us
W20260925 03:05:00.795964 24511 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 304288us
I20260925 03:05:00.728905 27061 raft_consensus.cc:2726] T c85f5d1d3f004cf5817bb594fd8e6469 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260925 03:05:00.757886 27065 log.cc:924] Time spent T 29637f5a4a504ce6ba29033ccd0cd716 P 1bc1f2e62ab44af995165bd5116a9d43: Append to log took a long time: real 0.438s user 0.000s sys 0.009s
I20260925 03:05:00.613183 27021 leader_election.cc:288] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:05:00.769182 26883 raft_consensus.cc:482] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:00.753979 26810 raft_consensus.cc:482] T 367277daf8d04778b3fa4ed94d71580d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:00.731317 27022 raft_consensus.cc:482] T 3d89396576884f7c8901dba54f783947 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:05:00.842242 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b5df303cd6a4d59aca5c18bbbff24ed" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 9 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:00.843194 25193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43"
I20260925 03:05:00.825199 27063 raft_consensus.cc:2726] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260925 03:05:00.830861 26979 raft_consensus.cc:482] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:00.757251 27067 raft_consensus.cc:3037] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Advancing to term 5
W20260925 03:05:00.868505 26682 consensus_meta.cc:287] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5: Time spent flushing consensus metadata: real 0.881s user 0.009s sys 0.003s
I20260925 03:05:00.891489 26682 raft_consensus.cc:504] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:00.905038 26979 raft_consensus.cc:504] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:05:00.906178 27073 log.cc:924] Time spent T cc2df40db08649c9a75e90000b878559 P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.327s user 0.000s sys 0.000s
I20260925 03:05:00.907632 26979 leader_election.cc:288] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:00.910027 27063 raft_consensus.cc:482] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:00.835186 26970 leader_election.cc:288] T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 8 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:05:00.890318 25193 raft_consensus.cc:3037] T 7e49248643df4131a1c42e74244e05c2 P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 FOLLOWER]: Advancing to term 8
I20260925 03:05:00.845892 27066 raft_consensus.cc:504] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:01.069875 26878 leader_election.cc:288] T ceab6b0fccd54e909a64f66fe81a9a1f P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:01.070612 27066 leader_election.cc:288] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:05:00.812479 26861 raft_consensus.cc:504] T 1414dcfe53f64a78ae413531eb8ec117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:01.072818 26861 leader_election.cc:288] T 1414dcfe53f64a78ae413531eb8ec117 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:01.073342 26897 leader_election.cc:288] T 37a157996e944db2906110d87ecb0c54 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:00.868929 26883 raft_consensus.cc:504] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:01.076099 26883 leader_election.cc:288] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:05:00.771082 26843 raft_consensus.cc:504] T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:00.922164 24547 raft_consensus.cc:1226] T a4abe0ea757e4886be4945c5001e529a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 LEADER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 2. Current term is 3. Ops: []
I20260925 03:05:00.910523 27063 raft_consensus.cc:504] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:00.817672 26900 consensus_queue.cc:1035] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.062s
I20260925 03:05:00.820928 26984 raft_consensus.cc:504] T 3d89396576884f7c8901dba54f783947 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:00.856037 27061 raft_consensus.cc:482] T 7993f9c3e2a44f408bbf5d1e395e2306 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:01.082984 24473 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1245) took 2649 ms. Trace:
I20260925 03:05:01.113910 24473 rpcz_store.cc:276] 0925 03:04:58.433391 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:59.136792 (+703401us) service_pool.cc:227] Handling call
0925 03:04:59.137109 (+ 317us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:05:00.020435 (+883326us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:05:01.082694 (+1062259us) spinlock_profiling.cc:243] Waited 839 ms on lock 0x7b5400119ed4. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d0f51 00007f43764cf99e 00007f437b4112e9 00007f4374af7684 00007f4374af7612 00007f4374af7541 00007f4374af74bc 00007f4374af6772 00007f43748fa94c 00007f43748f9d86
0925 03:05:01.082913 (+ 219us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1928646400}
W20260925 03:05:01.084870 25196 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.903s user 0.008s sys 0.016s
I20260925 03:05:00.832437 27077 raft_consensus.cc:482] T ac9357d144aa49789faaf31fe5e6bf89 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:01.066543 27055 consensus_queue.cc:237] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:00.911870 26810 raft_consensus.cc:504] T 367277daf8d04778b3fa4ed94d71580d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
W20260925 03:05:01.110242 24539 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.536s user 0.109s sys 0.128s
W20260925 03:05:01.134128 24539 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.560s user 0.109s sys 0.128s
I20260925 03:05:00.879400 27022 raft_consensus.cc:3037] T 3d89396576884f7c8901dba54f783947 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:05:01.087908 26878 raft_consensus.cc:482] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260925 03:05:01.115134 25196 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.944s user 0.008s sys 0.016s
I20260925 03:05:01.153568 25196 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49794 (request call id 264) took 1703 ms. Trace:
I20260925 03:05:00.878088 27021 consensus_queue.cc:1035] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.242s
I20260925 03:05:01.153982 25196 rpcz_store.cc:276] 0925 03:04:59.450618 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:59.457119 (+ 6501us) service_pool.cc:227] Handling call
0925 03:04:59.457440 (+ 321us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:59.457639 (+ 199us) raft_consensus.cc:1520] Early marking committed up to index 6
0925 03:04:59.617945 (+160306us) log.cc:842] Serialized 14 byte log entry
0925 03:05:01.153160 (+1535215us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:05:01.153225 (+ 65us) raft_consensus.cc:1631] Marking committed up to 6
0925 03:05:01.153279 (+ 54us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:05:01.153351 (+ 72us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:05:01.153487 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":23000,"prepare.queue_time_us":353745,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":41,"raft.queue_time_us":63669,"raft.run_cpu_time_us":148,"raft.run_wall_time_us":6910,"thread_start_us":1665292,"threads_started":3,"wal-append.queue_time_us":601774}
I20260925 03:05:00.844264 26863 consensus_queue.cc:1046] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.249s
W20260925 03:05:01.112332 24546 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.506s user 0.010s sys 0.046s
W20260925 03:05:01.159253 24546 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.553s user 0.010s sys 0.047s
I20260925 03:05:01.110991 26800 raft_consensus.cc:482] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:01.118105 26897 raft_consensus.cc:482] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:01.115676 27066 raft_consensus.cc:482] T 7b8e0d260a884ecdb20f79de46c36b19 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
W20260925 03:05:01.204965 25162 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.505s user 0.011s sys 0.011s
W20260925 03:05:01.205535 25162 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.542s user 0.011s sys 0.011s
W20260925 03:05:01.217029 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 385015us
I20260925 03:05:01.166357 27025 raft_consensus.cc:3037] T 29637f5a4a504ce6ba29033ccd0cd716 P 1bc1f2e62ab44af995165bd5116a9d43 [term 7 FOLLOWER]: Advancing to term 8
W20260925 03:05:01.205303 26937 consensus_meta.cc:287] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent flushing consensus metadata: real 0.887s user 0.006s sys 0.005s
I20260925 03:05:01.199476 26824 raft_consensus.cc:482] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:01.165217 26878 raft_consensus.cc:504] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:01.369175 27083 raft_consensus.cc:482] T 43366a0a2f9d4344bf4e37ffd290ef8a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
W20260925 03:05:01.370208 27074 consensus_meta.cc:287] T 04e068a0f80941dc9138cee063f53bab P eeab93abfdb141a895002ec9f312296c: Time spent flushing consensus metadata: real 0.793s user 0.012s sys 0.000s
W20260925 03:05:01.371272 27081 log.cc:924] Time spent T 367277daf8d04778b3fa4ed94d71580d P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.618s user 0.001s sys 0.000s
I20260925 03:05:01.379026 27074 raft_consensus.cc:504] T 04e068a0f80941dc9138cee063f53bab P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:01.192947 26897 raft_consensus.cc:504] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:01.331408 27095 raft_consensus.cc:482] T 11bb46bce3564841b7674e6a693bd2e6 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:01.214236 26800 raft_consensus.cc:504] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:01.135099 27077 raft_consensus.cc:504] T ac9357d144aa49789faaf31fe5e6bf89 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:01.219051 27066 raft_consensus.cc:504] T 7b8e0d260a884ecdb20f79de46c36b19 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:01.142426 27061 raft_consensus.cc:504] T 7993f9c3e2a44f408bbf5d1e395e2306 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:05:01.863430 24546 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.698s user 0.108s sys 0.239s
W20260925 03:05:01.887223 24546 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.722s user 0.109s sys 0.240s
I20260925 03:05:01.383399 27074 leader_election.cc:288] T 04e068a0f80941dc9138cee063f53bab P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:05:01.888832 27096 raft_consensus.cc:482] T e92f377a79354a6ebb4eec67303c13e3 P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:01.890337 27094 raft_consensus.cc:482] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260925 03:05:01.829944 25165 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.751s user 0.004s sys 0.032s
W20260925 03:05:01.893051 25165 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.814s user 0.005s sys 0.033s
W20260925 03:05:01.592217 24436 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.513s user 0.013s sys 0.012s
W20260925 03:05:01.894632 24436 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.816s user 0.013s sys 0.025s
W20260925 03:05:01.894912 24436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 1.02723e+06us
I20260925 03:05:01.637868 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9c33fd100ff44c1a3247319a324a044" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:01.898087 25196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e068a0f80941dc9138cee063f53bab" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43"
I20260925 03:05:01.903481 25196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:05:01.590523 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25102ba08e4e49388f6176badcd10a25" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
W20260925 03:05:01.536356 24511 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.702s user 0.066s sys 0.096s
W20260925 03:05:01.906100 24511 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.072s user 0.071s sys 0.127s
I20260925 03:05:01.907279 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b5df303cd6a4d59aca5c18bbbff24ed" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 9 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:01.908105 24541 raft_consensus.cc:2396] T 0b5df303cd6a4d59aca5c18bbbff24ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 for term 9 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260925 03:05:01.909073 24544 pending_rounds.cc:76] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Aborting all ops after (but not including) 3
I20260925 03:05:01.909415 24544 pending_rounds.cc:97] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20260925 03:05:01.909644 24544 raft_consensus.cc:2866] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260925 03:05:01.912248 24467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e5819c15eac4544a57e5ee6c9d4f117" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:01.912155 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e8af2641e124f9589c5ccfdac785f63" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:01.591197 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5f9b503b0ab43999d582015a367441e" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:01.368516 26824 raft_consensus.cc:504] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:01.889462 27096 raft_consensus.cc:504] T e92f377a79354a6ebb4eec67303c13e3 P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:01.890866 27094 raft_consensus.cc:504] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:01.918803 24464 raft_consensus.cc:2379] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c in current term 7: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
W20260925 03:05:02.210105 24513 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.880s user 0.007s sys 0.010s
W20260925 03:05:01.961834 24514 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.732s user 0.015s sys 0.015s
I20260925 03:05:02.178996 24466 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1255) took 3590 ms. Trace:
W20260925 03:05:02.563512 24514 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.334s user 0.015s sys 0.023s
W20260925 03:05:01.770020 24465 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.578s user 0.008s sys 0.030s
W20260925 03:05:02.561980 24511 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.650s user 0.009s sys 0.003s
W20260925 03:05:02.564388 24465 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.372s user 0.011s sys 0.036s
W20260925 03:05:02.564808 24511 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.653s user 0.010s sys 0.003s
I20260925 03:05:02.564978 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d89396576884f7c8901dba54f783947" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
W20260925 03:05:01.917106 24437 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.198s user 0.008s sys 0.021s
I20260925 03:05:02.566716 24464 raft_consensus.cc:2454] T 3d89396576884f7c8901dba54f783947 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 3.
W20260925 03:05:02.566560 24513 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.236s user 0.008s sys 0.011s
I20260925 03:05:02.005345 24467 raft_consensus.cc:2454] T 8e5819c15eac4544a57e5ee6c9d4f117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 7.
I20260925 03:05:02.568225 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c81a1b3b616492ab6c964f45af36077" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:02.569175 24464 raft_consensus.cc:2454] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 5.
I20260925 03:05:02.563746 24466 rpcz_store.cc:276] 0925 03:04:58.588279 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:59.513718 (+925439us) service_pool.cc:227] Handling call
0925 03:04:59.514645 (+ 927us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:04:59.670555 (+155910us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:59.670592 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:04:59.670634 (+ 42us) raft_consensus.cc:1631] Marking committed up to 5
0925 03:04:59.670666 (+ 32us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:05:02.178655 (+2507989us) spinlock_profiling.cc:243] Waited 156 ms on lock 0x7b54001199d4. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d0f51 00007f43764cf99e 00007f437b4112e9 00007f4374af7684 00007f4374af7612 00007f4374af7541 00007f4374af74bc 00007f4374af6772 00007f43748fa94c 00007f43748f9d86
0925 03:05:02.178721 (+ 66us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:05:02.178914 (+ 193us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":358074624}
I20260925 03:05:02.570348 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:02.571393 24464 raft_consensus.cc:3037] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Advancing to term 8
W20260925 03:05:02.566978 24437 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.847s user 0.008s sys 0.029s
W20260925 03:05:02.187358 24542 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.880s user 0.002s sys 0.048s
W20260925 03:05:02.573743 24542 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.336s user 0.002s sys 0.056s
I20260925 03:05:02.574026 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15861cd28e354598a3c6bfc4337c34bb" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:02.575255 24542 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 1184) took 2263 ms. Trace:
I20260925 03:05:01.916877 24472 raft_consensus.cc:2454] T 9e8af2641e124f9589c5ccfdac785f63 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 6.
I20260925 03:05:02.575804 24542 rpcz_store.cc:276] 0925 03:05:00.311662 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:05:00.410168 (+ 98506us) service_pool.cc:227] Handling call
0925 03:05:00.439541 (+ 29373us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:05:00.439768 (+ 227us) raft_consensus.cc:1520] Early marking committed up to index 4
0925 03:05:00.814424 (+374656us) log.cc:842] Serialized 14 byte log entry
0925 03:05:02.574331 (+1759907us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:05:02.574506 (+ 175us) raft_consensus.cc:1631] Marking committed up to 4
0925 03:05:02.574678 (+ 172us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:05:02.574828 (+ 150us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:05:02.575102 (+ 274us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":110970,"raft.queue_time_us":949753,"raft.run_cpu_time_us":194,"raft.run_wall_time_us":6681,"thread_start_us":1996813,"threads_started":3,"wal-append.queue_time_us":1049122}
I20260925 03:05:02.569366 24467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442d4a29b1d743599f4b6b4871e26668" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:02.578369 24467 raft_consensus.cc:3037] T 442d4a29b1d743599f4b6b4871e26668 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:05:02.578768 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "293ef69caf514ac98510a68495f54eda" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:02.580345 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367277daf8d04778b3fa4ed94d71580d" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:02.581480 24542 raft_consensus.cc:2396] T 367277daf8d04778b3fa4ed94d71580d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1bc1f2e62ab44af995165bd5116a9d43 for term 7 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20260925 03:05:02.566169 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06f4a7eaf6534e4986c9fd35911038e4" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:02.582652 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
W20260925 03:05:02.583250 24544 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.673s user 0.009s sys 0.003s
W20260925 03:05:02.590251 24544 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.680s user 0.009s sys 0.003s
W20260925 03:05:01.916949 25163 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.837s user 0.012s sys 0.016s
W20260925 03:05:02.591565 25163 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.512s user 0.018s sys 0.026s
W20260925 03:05:02.583801 24545 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.676s user 0.003s sys 0.004s
W20260925 03:05:02.583806 24546 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.605s user 0.000s sys 0.007s
W20260925 03:05:01.917483 24586 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.442s user 0.000s sys 0.029s
I20260925 03:05:02.127446 24618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d89396576884f7c8901dba54f783947" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:01.986964 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5aec38e66a947c681e6c2a72b05e6ca" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
W20260925 03:05:01.383529 24587 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.672s user 0.009s sys 0.013s
I20260925 03:05:01.987303 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1414dcfe53f64a78ae413531eb8ec117" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:01.985072 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceab6b0fccd54e909a64f66fe81a9a1f" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:05:01.986603 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37a157996e944db2906110d87ecb0c54" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:02.018141 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
W20260925 03:05:02.205076 25162 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.877s user 0.015s sys 0.036s
I20260925 03:05:02.583998 24470 raft_consensus.cc:1226] T 43366a0a2f9d4344bf4e37ffd290ef8a P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 3. Current term is 4. Ops: []
I20260925 03:05:01.937448 25196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5aec38e66a947c681e6c2a72b05e6ca" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:05:02.449481 25194 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48408 (request call id 214) took 2366 ms. Trace:
W20260925 03:05:02.351306 26956 consensus_meta.cc:287] T 21c43af5e6844fdaacc8c625fa01fd8d P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent flushing consensus metadata: real 1.622s user 0.014s sys 0.000s
I20260925 03:05:02.612483 27129 raft_consensus.cc:482] T 100c113a9fd84cc8adf2ec7b33ea0093 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:02.218562 26800 leader_election.cc:288] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
W20260925 03:05:01.537998 26829 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.705s user 0.038s sys 0.124s
I20260925 03:05:02.616149 27129 raft_consensus.cc:504] T 100c113a9fd84cc8adf2ec7b33ea0093 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:05:02.603384 24436 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.695s user 0.008s sys 0.000s
I20260925 03:05:02.603930 27125 raft_consensus.cc:482] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:02.605329 27137 raft_consensus.cc:482] T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1bc1f2e62ab44af995165bd5116a9d43)
W20260925 03:05:01.383348 27091 log.cc:924] Time spent T a4abe0ea757e4886be4945c5001e529a P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.056s user 0.000s sys 0.000s
I20260925 03:05:02.588235 24469 raft_consensus.cc:1261] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
W20260925 03:05:02.090824 25193 consensus_meta.cc:287] T 7e49248643df4131a1c42e74244e05c2 P 1bc1f2e62ab44af995165bd5116a9d43: Time spent flushing consensus metadata: real 1.023s user 0.008s sys 0.004s
I20260925 03:05:02.106364 26878 leader_election.cc:288] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:02.609784 27122 raft_consensus.cc:482] T b030d84c670b48d5bff337ccc0c77626 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:01.375181 26754 leader_election.cc:288] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 7 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:05:02.610749 27124 raft_consensus.cc:482] T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:02.611591 27123 raft_consensus.cc:482] T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
W20260925 03:05:02.618212 24545 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.710s user 0.004s sys 0.004s
W20260925 03:05:02.308561 27024 consensus_meta.cc:287] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43: Time spent flushing consensus metadata: real 1.921s user 0.012s sys 0.000s
W20260925 03:05:01.959739 24588 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.835s user 0.012s sys 0.019s
I20260925 03:05:02.617547 26800 raft_consensus.cc:482] T c85f5d1d3f004cf5817bb594fd8e6469 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
W20260925 03:05:01.962827 27022 consensus_meta.cc:287] T 3d89396576884f7c8901dba54f783947 P eeab93abfdb141a895002ec9f312296c: Time spent flushing consensus metadata: real 0.814s user 0.010s sys 0.000s
W20260925 03:05:01.532323 26874 consensus_meta.cc:287] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5: Time spent flushing consensus metadata: real 1.122s user 0.005s sys 0.004s
W20260925 03:05:01.916829 26923 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.513s user 0.007s sys 0.021s
W20260925 03:05:01.481086 26913 consensus_meta.cc:287] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5: Time spent flushing consensus metadata: real 1.629s user 0.012s sys 0.002s
I20260925 03:05:01.331889 26937 raft_consensus.cc:2726] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260925 03:05:02.377676 27035 thread.cc:636] apply [worker] (thread pool) Time spent creating pthread: real 2.518s user 0.008s sys 0.009s
W20260925 03:05:02.617985 24546 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.640s user 0.000s sys 0.008s
W20260925 03:05:01.881249 26954 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.459s user 0.019s sys 0.020s
W20260925 03:05:02.619863 24586 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 2.144s user 0.013s sys 0.058s
W20260925 03:05:02.612921 25162 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.399s user 0.025s sys 0.051s
I20260925 03:05:02.607949 24465 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38920 (request call id 164) took 2502 ms. Trace:
I20260925 03:05:02.614944 24470 raft_consensus.cc:1226] T 0be4e0aa984f4f2b81a58e382e434ad5 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 5. Current term is 6. Ops: []
W20260925 03:05:02.684314 24436 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.776s user 0.010s sys 0.000s
W20260925 03:05:02.617462 26829 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.789s user 0.050s sys 0.153s
I20260925 03:05:02.590185 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74443a0a1dc34d88a2f42e3b722db405" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
W20260925 03:05:02.677270 25163 connection.cc:232] RPC call timeout handler was delayed by -1.3691180789992359s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1515s for a response
W20260925 03:05:02.240136 26882 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.879s user 0.002s sys 0.015s
W20260925 03:05:01.382320 27082 log.cc:924] Time spent T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.254s user 0.001s sys 0.000s
W20260925 03:05:01.929678 26756 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.858s user 0.010s sys 0.018s
I20260925 03:05:02.782232 27136 raft_consensus.cc:482] T 8e5819c15eac4544a57e5ee6c9d4f117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:02.783155 27136 raft_consensus.cc:504] T 8e5819c15eac4544a57e5ee6c9d4f117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:05:02.784852 24588 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.654s user 0.022s sys 0.028s
I20260925 03:05:02.785284 27058 raft_consensus.cc:2726] T f22ddd7d1c444dc58f45a1fb0b3e02bc P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260925 03:05:02.785105 27024 raft_consensus.cc:504] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
W20260925 03:05:02.782267 24587 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 2.321s user 0.011s sys 0.048s
W20260925 03:05:01.985993 26980 consensus_meta.cc:287] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent flushing consensus metadata: real 1.145s user 0.007s sys 0.004s
W20260925 03:05:02.343107 26947 consensus_meta.cc:287] T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5: Time spent flushing consensus metadata: real 1.896s user 0.011s sys 0.001s
W20260925 03:05:02.224215 27078 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.840s user 0.005s sys 0.019s
W20260925 03:05:02.161319 27097 log.cc:924] Time spent T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.081s user 0.000s sys 0.000s
W20260925 03:05:02.777825 26923 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 2.380s user 0.018s sys 0.030s
W20260925 03:05:02.183632 27025 consensus_meta.cc:287] T 29637f5a4a504ce6ba29033ccd0cd716 P 1bc1f2e62ab44af995165bd5116a9d43: Time spent flushing consensus metadata: real 0.856s user 0.005s sys 0.006s
W20260925 03:05:01.381675 27072 log.cc:924] Time spent T 293ef69caf514ac98510a68495f54eda P 869b0a8575b54d08ae2817fad5dd42d5: Append to log took a long time: real 0.530s user 0.000s sys 0.000s
W20260925 03:05:02.776059 27035 thread.cc:602] apply [worker] (thread pool) Time spent starting thread: real 2.918s user 0.011s sys 0.013s
I20260925 03:05:02.776376 27125 raft_consensus.cc:504] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:02.777243 24546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 1191) took 2413 ms. Trace:
W20260925 03:05:02.302008 27103 log.cc:924] Time spent T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.106s user 0.000s sys 0.000s
W20260925 03:05:02.208966 27099 log.cc:924] Time spent T f22ddd7d1c444dc58f45a1fb0b3e02bc P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.321s user 0.000s sys 0.001s
I20260925 03:05:01.855577 27021 raft_consensus.cc:482] T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:02.772595 24469 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1261) took 1121 ms. Trace:
I20260925 03:05:02.163579 27107 raft_consensus.cc:482] T a7aa47c48404465b94a36fd819fe3c7a P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:02.775964 27137 raft_consensus.cc:504] T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
W20260925 03:05:02.781849 26954 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 2.368s user 0.032s sys 0.036s
I20260925 03:05:02.781443 27022 raft_consensus.cc:504] T 3d89396576884f7c8901dba54f783947 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:02.287495 27118 raft_consensus.cc:482] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:01.962237 27101 raft_consensus.cc:482] T 5278a60f97da4e10aecd50a8d9a9a43c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:02.160907 27066 leader_election.cc:288] T 7b8e0d260a884ecdb20f79de46c36b19 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:01.338514 26861 raft_consensus.cc:482] T 21c43af5e6844fdaacc8c625fa01fd8d P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:02.793371 25193 raft_consensus.cc:2454] T 7e49248643df4131a1c42e74244e05c2 P 1bc1f2e62ab44af995165bd5116a9d43 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 8.
I20260925 03:05:02.794037 27138 raft_consensus.cc:482] T 0b5df303cd6a4d59aca5c18bbbff24ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:02.254774 27102 raft_consensus.cc:482] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:02.026458 26970 raft_consensus.cc:482] T 550118bc1a0641ce9803bdbf43f2069f P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:05:02.153982 27093 raft_consensus.cc:482] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:02.206836 27104 raft_consensus.cc:482] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:02.301426 27116 raft_consensus.cc:482] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:01.624047 26984 leader_election.cc:288] T 3d89396576884f7c8901dba54f783947 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:02.018432 26810 leader_election.cc:288] T 367277daf8d04778b3fa4ed94d71580d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:05:02.796289 27096 leader_election.cc:288] T e92f377a79354a6ebb4eec67303c13e3 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:02.990183 27024 leader_election.cc:288] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 6 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:05:02.989850 26861 raft_consensus.cc:504] T 21c43af5e6844fdaacc8c625fa01fd8d P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:02.774230 24465 rpcz_store.cc:276] 0925 03:05:00.105152 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:05:00.105912 (+ 760us) service_pool.cc:227] Handling call
0925 03:05:00.148326 (+ 42414us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:05:00.148547 (+ 221us) raft_consensus.cc:1520] Early marking committed up to index 6
0925 03:05:00.192182 (+ 43635us) log.cc:842] Serialized 14 byte log entry
0925 03:05:02.607225 (+2415043us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:05:02.607353 (+ 128us) raft_consensus.cc:1631] Marking committed up to 6
0925 03:05:02.607479 (+ 126us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:05:02.607611 (+ 132us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:05:02.607828 (+ 217us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":43520,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":39,"thread_start_us":2414157,"threads_started":2,"wal-append.queue_time_us":2004197}
I20260925 03:05:01.635591 26843 leader_election.cc:288] T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:02.084467 27110 raft_consensus.cc:482] T b627ecb4a5384ef392aa695e06599c39 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:02.796168 27123 raft_consensus.cc:504] T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:05:03.000253 25193 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49794 (request call id 267) took 2157 ms (client timeout 2867 ms). Trace:
W20260925 03:05:03.000849 25193 rpcz_store.cc:269] 0925 03:05:00.842677 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:05:00.842865 (+ 188us) service_pool.cc:227] Handling call
0925 03:05:03.000079 (+2157214us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:05:03.001705 27093 raft_consensus.cc:504] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:03.002245 27138 raft_consensus.cc:504] T 0b5df303cd6a4d59aca5c18bbbff24ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:01.336522 27067 raft_consensus.cc:504] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:03.005182 27102 raft_consensus.cc:504] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:03.007140 27067 leader_election.cc:288] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:05:03.008462 27019 raft_consensus.cc:482] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:03.008988 25196 raft_consensus.cc:2454] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 9.
I20260925 03:05:03.009588 26824 leader_election.cc:288] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:02.797043 27135 raft_consensus.cc:482] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:03.010267 26861 leader_election.cc:288] T 21c43af5e6844fdaacc8c625fa01fd8d P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:03.011050 27135 raft_consensus.cc:504] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:02.992165 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4abe0ea757e4886be4945c5001e529a" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:01.379570 27083 raft_consensus.cc:504] T 43366a0a2f9d4344bf4e37ffd290ef8a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:02.161404 27077 leader_election.cc:288] T ac9357d144aa49789faaf31fe5e6bf89 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:02.189352 27061 leader_election.cc:288] T 7993f9c3e2a44f408bbf5d1e395e2306 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:01.383584 27063 leader_election.cc:288] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
W20260925 03:05:02.110026 27014 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.390s user 0.000s sys 0.028s
I20260925 03:05:02.968458 24622 raft_consensus.cc:2379] T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 in current term 8: Already voted for candidate eeab93abfdb141a895002ec9f312296c in this term.
W20260925 03:05:02.909086 24439 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.819s user 0.023s sys 0.036s
W20260925 03:05:02.971400 24473 consensus_meta.cc:287] T 25102ba08e4e49388f6176badcd10a25 P 869b0a8575b54d08ae2817fad5dd42d5: Time spent flushing consensus metadata: real 1.066s user 0.017s sys 0.003s
W20260925 03:05:02.968932 24438 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.696s user 0.059s sys 0.086s
I20260925 03:05:02.161059 27105 raft_consensus.cc:482] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:02.339762 27115 raft_consensus.cc:482] T 7993f9c3e2a44f408bbf5d1e395e2306 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:02.993192 24469 rpcz_store.cc:276] 0925 03:05:01.648613 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:05:01.648765 (+ 152us) service_pool.cc:227] Handling call
0925 03:05:02.587115 (+938350us) spinlock_profiling.cc:243] Waited 15.1 ms on lock 0x7b5800001598. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f43733ecf11 00007f4373570851 00007f4373570702 00007f437c460b09 00007f437b39c730 00007f437b46279b 00007f437b462821 00007f437b4262b0 00007f437b4117f9 00007f437b4112a2 00007f4374af7684
0925 03:05:02.587574 (+ 459us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:05:02.714808 (+127234us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:05:02.769762 (+ 54954us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":34762112}
W20260925 03:05:02.980804 27078 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.526s user 0.007s sys 0.026s
W20260925 03:05:02.772538 24511 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 166142us
W20260925 03:05:02.988766 26882 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.490s user 0.009s sys 0.021s
I20260925 03:05:02.616195 26956 raft_consensus.cc:504] T 21c43af5e6844fdaacc8c625fa01fd8d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:03.007862 27067 raft_consensus.cc:482] T 0be4e0aa984f4f2b81a58e382e434ad5 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:03.068863 24622 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38570 (request call id 1584) took 1084 ms. Trace:
I20260925 03:05:03.069239 24622 rpcz_store.cc:276] 0925 03:05:01.984411 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:05:01.985270 (+ 859us) service_pool.cc:227] Handling call
0925 03:05:03.060613 (+1075343us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:05:03.069682 26970 raft_consensus.cc:504] T 550118bc1a0641ce9803bdbf43f2069f P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:03.072350 26970 leader_election.cc:288] T 550118bc1a0641ce9803bdbf43f2069f P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
W20260925 03:05:02.884845 26756 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.784s user 0.028s sys 0.066s
I20260925 03:05:03.038856 24467 raft_consensus.cc:2454] T 442d4a29b1d743599f4b6b4871e26668 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 7.
I20260925 03:05:03.004485 27093 leader_election.cc:288] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:02.970033 26937 raft_consensus.cc:482] T 37a157996e944db2906110d87ecb0c54 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:03.049162 27116 raft_consensus.cc:504] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:05:03.050467 27014 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.806s user 0.003s sys 0.054s
I20260925 03:05:03.051294 27063 raft_consensus.cc:482] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:03.052785 27136 leader_election.cc:288] T 8e5819c15eac4544a57e5ee6c9d4f117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:03.007768 27102 leader_election.cc:288] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:03.053545 24464 raft_consensus.cc:2454] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 8.
W20260925 03:05:03.054426 24439 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.964s user 0.026s sys 0.044s
I20260925 03:05:03.048377 26682 leader_election.cc:288] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
W20260925 03:05:03.055596 24438 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.984s user 0.059s sys 0.088s
I20260925 03:05:03.061805 26861 raft_consensus.cc:482] T b4ebd2b043024a548005c8b6b00b0449 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1bc1f2e62ab44af995165bd5116a9d43)
I20260925 03:05:03.066427 26749 consensus_queue.cc:1046] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true }, Status: INVALID_TERM, Last received: 7.6, Next index: 7, Last known committed idx: 4, Time since last communication: 0.001s
I20260925 03:05:03.067402 24473 raft_consensus.cc:2454] T 25102ba08e4e49388f6176badcd10a25 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 5.
I20260925 03:05:03.068529 27067 raft_consensus.cc:504] T 0be4e0aa984f4f2b81a58e382e434ad5 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:03.073828 27123 leader_election.cc:288] T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:01.963068 27095 raft_consensus.cc:504] T 11bb46bce3564841b7674e6a693bd2e6 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:03.076347 27104 raft_consensus.cc:504] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:03.061599 26954 raft_consensus.cc:482] T 0be4e0aa984f4f2b81a58e382e434ad5 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:03.082026 27129 leader_election.cc:288] T 100c113a9fd84cc8adf2ec7b33ea0093 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:03.084167 27083 leader_election.cc:288] T 43366a0a2f9d4344bf4e37ffd290ef8a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:03.059556 24541 raft_consensus.cc:2454] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 5.
I20260925 03:05:03.085129 27110 raft_consensus.cc:504] T b627ecb4a5384ef392aa695e06599c39 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:03.085883 26900 raft_consensus.cc:482] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260925 03:05:03.034209 25163 connection.cc:232] RPC call timeout handler was delayed by -1.1194300829974964s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1738s for a response
I20260925 03:05:02.991510 26810 raft_consensus.cc:482] T 48ca6ccd98194cf2af7b5278d15bf4ea P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:02.991871 24546 rpcz_store.cc:276] 0925 03:05:00.363828 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:05:00.453348 (+ 89520us) service_pool.cc:227] Handling call
0925 03:05:00.453800 (+ 452us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:05:00.454005 (+ 205us) raft_consensus.cc:1520] Early marking committed up to index 3
0925 03:05:01.165323 (+711318us) log.cc:842] Serialized 14 byte log entry
0925 03:05:02.776484 (+1611161us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:05:02.776631 (+ 147us) raft_consensus.cc:1631] Marking committed up to 3
0925 03:05:02.776761 (+ 130us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:05:02.776860 (+ 99us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:05:02.777158 (+ 298us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":216030,"raft.queue_time_us":413290,"raft.run_cpu_time_us":99,"raft.run_wall_time_us":100,"thread_start_us":2069888,"threads_started":3,"wal-append.queue_time_us":1120175}
I20260925 03:05:03.097885 24467 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 1268) took 1438 ms. Trace:
I20260925 03:05:03.102676 24618 raft_consensus.cc:2354] T 3d89396576884f7c8901dba54f783947 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 4. Current term is 5.
I20260925 03:05:02.781339 26874 raft_consensus.cc:504] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:03.106509 24464 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 1274) took 1069 ms. Trace:
I20260925 03:05:02.997740 26800 raft_consensus.cc:504] T c85f5d1d3f004cf5817bb594fd8e6469 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:05:03.123425 24473 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 1257) took 1774 ms (client timeout 2210 ms). Trace:
I20260925 03:05:03.123131 27115 raft_consensus.cc:504] T 7993f9c3e2a44f408bbf5d1e395e2306 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:03.124140 26937 raft_consensus.cc:504] T 37a157996e944db2906110d87ecb0c54 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:03.125169 27107 raft_consensus.cc:504] T a7aa47c48404465b94a36fd819fe3c7a P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:03.009244 27019 raft_consensus.cc:504] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:03.086728 27135 leader_election.cc:288] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:05:03.096029 27138 leader_election.cc:288] T 0b5df303cd6a4d59aca5c18bbbff24ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:03.021704 27124 raft_consensus.cc:504] T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:03.126133 27105 raft_consensus.cc:504] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:02.995353 27122 raft_consensus.cc:504] T b030d84c670b48d5bff337ccc0c77626 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:05:03.127691 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 127609us
I20260925 03:05:03.202937 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b8e0d260a884ecdb20f79de46c36b19" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:03.243731 24464 rpcz_store.cc:276] 0925 03:05:02.037449 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:05:02.569960 (+532511us) service_pool.cc:227] Handling call
0925 03:05:03.106440 (+536480us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:05:03.200398 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ac466793d094325ace224708b2158ed" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:03.162566 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e068a0f80941dc9138cee063f53bab" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:03.129379 24467 rpcz_store.cc:276] 0925 03:05:01.650653 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:05:02.568882 (+918229us) service_pool.cc:227] Handling call
0925 03:05:03.089312 (+520430us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:05:03.241617 27067 leader_election.cc:288] T 0be4e0aa984f4f2b81a58e382e434ad5 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
W20260925 03:05:03.129210 25163 connection.cc:232] RPC call timeout handler was delayed by -1.0691065079990949s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1748s for a response
W20260925 03:05:03.259927 24473 rpcz_store.cc:269] 0925 03:05:01.589935 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:05:01.590168 (+ 233us) service_pool.cc:227] Handling call
0925 03:05:03.120945 (+1530777us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260925 03:05:03.259361 25163 connection.cc:232] RPC call timeout handler was delayed by -1.0326358499987691s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19130000000000003s for a response
W20260925 03:05:03.263671 25163 connection.cc:232] RPC call timeout handler was delayed by -1.0146823509985552s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1756s for a response
I20260925 03:05:03.203513 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5aec38e66a947c681e6c2a72b05e6ca" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:03.233974 27118 raft_consensus.cc:504] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:03.264452 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ac285175464fbda67211d9574db415" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:03.198051 26682 raft_consensus.cc:482] T 100c113a9fd84cc8adf2ec7b33ea0093 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:03.200945 27134 raft_consensus.cc:482] T e6b6ec010caf4689b8c90449764f5cf9 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:03.301434 26937 leader_election.cc:288] T 37a157996e944db2906110d87ecb0c54 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:03.198612 26985 raft_consensus.cc:482] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:03.221925 26956 leader_election.cc:288] T 21c43af5e6844fdaacc8c625fa01fd8d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:03.198794 27025 raft_consensus.cc:2726] T 29637f5a4a504ce6ba29033ccd0cd716 P 1bc1f2e62ab44af995165bd5116a9d43 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260925 03:05:03.305886 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9357d144aa49789faaf31fe5e6bf89" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:03.204217 26955 raft_consensus.cc:482] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:03.305920 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e92f377a79354a6ebb4eec67303c13e3" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:03.301342 27124 leader_election.cc:288] T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:05:03.243904 26861 raft_consensus.cc:504] T b4ebd2b043024a548005c8b6b00b0449 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:03.300537 27125 leader_election.cc:288] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:03.320160 27048 raft_consensus.cc:482] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:05:03.318188 24586 leader_election.cc:302] T 442d4a29b1d743599f4b6b4871e26668 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:05:03.311694 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6b6ec010caf4689b8c90449764f5cf9" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:03.252897 27022 leader_election.cc:288] T 3d89396576884f7c8901dba54f783947 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:05:03.322570 24546 raft_consensus.cc:2379] T 8ac466793d094325ace224708b2158ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1bc1f2e62ab44af995165bd5116a9d43 in current term 6: Already voted for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in this term.
I20260925 03:05:03.199529 27021 raft_consensus.cc:504] T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:03.203277 26954 raft_consensus.cc:504] T 0be4e0aa984f4f2b81a58e382e434ad5 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:03.301601 27134 raft_consensus.cc:504] T e6b6ec010caf4689b8c90449764f5cf9 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:03.323035 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7993f9c3e2a44f408bbf5d1e395e2306" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:03.305188 27137 leader_election.cc:288] T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:05:03.320927 27048 raft_consensus.cc:504] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:03.327301 27115 leader_election.cc:288] T 7993f9c3e2a44f408bbf5d1e395e2306 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:03.328068 27126 raft_consensus.cc:482] T c88cbd39e01f44628a094af516be2833 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:05:03.199971 26810 raft_consensus.cc:504] T 48ca6ccd98194cf2af7b5278d15bf4ea P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:03.300513 26682 raft_consensus.cc:504] T 100c113a9fd84cc8adf2ec7b33ea0093 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:03.332018 25197 raft_consensus.cc:1261] T a4abe0ea757e4886be4945c5001e529a P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 FOLLOWER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260925 03:05:03.315735 26980 raft_consensus.cc:504] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:03.225065 27063 raft_consensus.cc:504] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:03.199343 26947 raft_consensus.cc:504] T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:03.201143 26900 raft_consensus.cc:504] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:03.262176 26897 leader_election.cc:288] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:03.199708 27101 raft_consensus.cc:504] T 5278a60f97da4e10aecd50a8d9a9a43c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:03.328742 26955 raft_consensus.cc:504] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:03.337786 27135 raft_consensus.cc:482] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:03.298854 27083 raft_consensus.cc:482] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:03.359929 24622 raft_consensus.cc:2454] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 6.
I20260925 03:05:03.347513 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71f461a6e2ff434993b78a2d077f0abd" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:02.615041 25194 rpcz_store.cc:276] 0925 03:05:00.083109 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:05:00.084706 (+ 1597us) service_pool.cc:227] Handling call
0925 03:05:00.903490 (+818784us) spinlock_profiling.cc:243] Waited 10.2 ms on lock 0x7b5800004e98. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f43733ecf11 00007f4373570851 00007f4373570bce 000000000034aea9 00007f437b39ca30 00007f437b4627b4 00007f437b462821 00007f437b4262b0 00007f437b4117f9 00007f437b4112a2 00007f4374af7684
0925 03:05:02.186102 (+1282612us) spinlock_profiling.cc:243] Waited 28.5 ms on lock 0x7b5800004e98. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f43733ecf11 00007f4373570c6c 00007f4373570bd6 000000000034aea9 00007f437b39ca30 00007f437b4627b4 00007f437b462821 00007f437b4262b0 00007f437b4117f9 00007f437b4112a2 00007f4374af7684
0925 03:05:02.188107 (+ 2005us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:05:02.198119 (+ 10012us) raft_consensus.cc:1520] Early marking committed up to index 8
0925 03:05:02.357113 (+158994us) log.cc:842] Serialized 14 byte log entry
0925 03:05:02.449042 (+ 91929us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:05:02.449105 (+ 63us) raft_consensus.cc:1631] Marking committed up to 8
0925 03:05:02.449155 (+ 50us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:05:02.449234 (+ 79us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:05:02.449388 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":152865,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":46,"raft.queue_time_us":87,"raft.run_cpu_time_us":119,"raft.run_wall_time_us":119,"spinlock_wait_cycles":88941056,"thread_start_us":162199,"threads_started":2,"wal-append.queue_time_us":89109}
I20260925 03:05:03.320291 27105 leader_election.cc:288] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:03.345700 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:03.346801 24470 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1275) took 1255 ms. Trace:
I20260925 03:05:03.333019 27125 raft_consensus.cc:482] T c85f5d1d3f004cf5817bb594fd8e6469 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:03.359978 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21c43af5e6844fdaacc8c625fa01fd8d" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:03.361817 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15861cd28e354598a3c6bfc4337c34bb" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:03.335112 27124 raft_consensus.cc:482] T a7aa47c48404465b94a36fd819fe3c7a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:03.367983 24464 raft_consensus.cc:2379] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 1bc1f2e62ab44af995165bd5116a9d43 in current term 6: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:05:03.364357 24546 raft_consensus.cc:2454] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 8.
I20260925 03:05:03.369472 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f68804eb5a244d87bb395d50683c6012" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:03.354315 27123 raft_consensus.cc:482] T 25102ba08e4e49388f6176badcd10a25 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:03.332439 26985 raft_consensus.cc:504] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:03.377394 24542 raft_consensus.cc:2354] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1bc1f2e62ab44af995165bd5116a9d43 for earlier term 7. Current term is 8.
I20260925 03:05:03.379622 25191 raft_consensus.cc:1226] T 29637f5a4a504ce6ba29033ccd0cd716 P 1bc1f2e62ab44af995165bd5116a9d43 [term 8 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 7. Current term is 8. Ops: []
I20260925 03:05:03.386297 26947 leader_election.cc:288] T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:03.387321 24613 raft_consensus.cc:2379] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 in current term 7: Already voted for candidate eeab93abfdb141a895002ec9f312296c in this term.
I20260925 03:05:03.378718 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367277daf8d04778b3fa4ed94d71580d" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:03.370460 27145 raft_consensus.cc:2781] T 442d4a29b1d743599f4b6b4871e26668 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Leader election won for term 7
I20260925 03:05:03.370777 26874 leader_election.cc:288] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:05:03.393555 27147 raft_consensus.cc:482] T c88cbd39e01f44628a094af516be2833 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:03.394057 27125 raft_consensus.cc:504] T c85f5d1d3f004cf5817bb594fd8e6469 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:03.364289 27122 leader_election.cc:288] T b030d84c670b48d5bff337ccc0c77626 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:03.370046 27126 raft_consensus.cc:504] T c88cbd39e01f44628a094af516be2833 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:03.371505 27124 raft_consensus.cc:504] T a7aa47c48404465b94a36fd819fe3c7a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:05:03.374541 27008 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 2.470s user 0.041s sys 0.087s
I20260925 03:05:03.490978 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ac466793d094325ace224708b2158ed" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:05:03.491513 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5aec38e66a947c681e6c2a72b05e6ca" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
W20260925 03:05:03.393288 27141 log.cc:924] Time spent T 1414dcfe53f64a78ae413531eb8ec117 P 869b0a8575b54d08ae2817fad5dd42d5: Append to log took a long time: real 0.090s user 0.000s sys 0.000s
I20260925 03:05:03.441351 27095 leader_election.cc:288] T 11bb46bce3564841b7674e6a693bd2e6 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:03.461796 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:03.387360 27135 raft_consensus.cc:504] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:03.504398 24546 raft_consensus.cc:2379] T 8ac466793d094325ace224708b2158ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 1bc1f2e62ab44af995165bd5116a9d43 in current term 6: Already voted for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in this term.
I20260925 03:05:03.364996 24470 rpcz_store.cc:276] 0925 03:05:02.086664 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:05:02.785091 (+698427us) service_pool.cc:227] Handling call
0925 03:05:03.078560 (+293469us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:05:03.078998 (+ 438us) raft_consensus.cc:1520] Early marking committed up to index 4
0925 03:05:03.132532 (+ 53534us) log.cc:842] Serialized 14 byte log entry
0925 03:05:03.341148 (+208616us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:05:03.341209 (+ 61us) raft_consensus.cc:1631] Marking committed up to 4
0925 03:05:03.341256 (+ 47us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:05:03.341324 (+ 68us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:05:03.341451 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":277351,"raft.queue_time_us":5519,"thread_start_us":159606,"threads_started":2}
W20260925 03:05:03.395783 25165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 48126.4us
I20260925 03:05:03.465840 24544 raft_consensus.cc:2454] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 7.
I20260925 03:05:03.463323 24539 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 1524) took 3069 ms. Trace:
I20260925 03:05:03.518038 24621 raft_consensus.cc:1261] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260925 03:05:03.518208 26811 raft_consensus.cc:482] T edcf6e6a422d4d198d4fbb0f7d3d845d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:03.519191 26810 leader_election.cc:288] T 48ca6ccd98194cf2af7b5278d15bf4ea P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
W20260925 03:05:03.506117 27008 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.669s user 0.052s sys 0.093s
I20260925 03:05:03.525167 24613 raft_consensus.cc:2379] T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 8: Already voted for candidate eeab93abfdb141a895002ec9f312296c in this term.
I20260925 03:05:03.497423 27083 raft_consensus.cc:504] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:03.519749 27147 raft_consensus.cc:504] T c88cbd39e01f44628a094af516be2833 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:03.538682 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e54ca1757ee4e42956a9cebbeca4f62" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:03.538225 27021 leader_election.cc:288] T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:03.478338 27142 raft_consensus.cc:482] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:03.520941 25163 leader_election.cc:302] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
W20260925 03:05:03.541687 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 53105.8us
W20260925 03:05:03.535490 24511 connection.cc:232] RPC call timeout handler was delayed by -1.4139395879992662s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1935s for a response
I20260925 03:05:03.488309 27123 raft_consensus.cc:504] T 25102ba08e4e49388f6176badcd10a25 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:03.543730 24619 raft_consensus.cc:2454] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 9.
I20260925 03:05:03.505275 27145 raft_consensus.cc:686] T 442d4a29b1d743599f4b6b4871e26668 P eeab93abfdb141a895002ec9f312296c [term 7 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:05:03.544546 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35992316cb8d4c028333388eff02ecde" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:05:03.516369 26954 leader_election.cc:288] T 0be4e0aa984f4f2b81a58e382e434ad5 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:03.507642 27105 consensus_queue.cc:1035] T b4ebd2b043024a548005c8b6b00b0449 P 1bc1f2e62ab44af995165bd5116a9d43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.029s
I20260925 03:05:03.550371 24464 raft_consensus.cc:2454] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 9.
I20260925 03:05:03.551342 24544 raft_consensus.cc:1261] T 8ac466793d094325ace224708b2158ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Refusing update from remote peer 1bc1f2e62ab44af995165bd5116a9d43: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260925 03:05:03.533505 26811 raft_consensus.cc:504] T edcf6e6a422d4d198d4fbb0f7d3d845d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:03.551700 25162 leader_election.cc:302] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:05:03.524108 24539 rpcz_store.cc:276] 0925 03:05:00.393974 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:05:00.475313 (+ 81339us) service_pool.cc:227] Handling call
0925 03:05:00.475675 (+ 362us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:05:00.476027 (+ 352us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:05:00.476053 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:05:00.542153 (+ 66100us) log.cc:842] Serialized 26 byte log entry
0925 03:05:01.134582 (+592429us) raft_consensus.cc:1631] Marking committed up to 4
0925 03:05:03.130451 (+1995869us) log.cc:842] Serialized 14 byte log entry
0925 03:05:03.440400 (+309949us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:05:03.462913 (+ 22513us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:05:03.462990 (+ 77us) raft_consensus.cc:1671] finished
0925 03:05:03.463001 (+ 11us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:05:03.463153 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1907674,"prepare.queue_time_us":1967378,"prepare.run_cpu_time_us":23,"prepare.run_wall_time_us":24,"raft.queue_time_us":79782,"raft.run_cpu_time_us":287,"raft.run_wall_time_us":16679,"thread_start_us":713791,"threads_started":3,"wal-append.queue_time_us":823579,"wal-append.run_cpu_time_us":2823,"wal-append.run_wall_time_us":1002833}
I20260925 03:05:03.562866 27143 raft_consensus.cc:2726] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260925 03:05:03.538782 27048 leader_election.cc:288] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:05:03.563333 24472 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1269) took 1888 ms. Trace:
I20260925 03:05:03.543223 26682 leader_election.cc:288] T 100c113a9fd84cc8adf2ec7b33ea0093 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
W20260925 03:05:03.573612 24540 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 1506) took 4781 ms (client timeout 2735 ms). Trace:
I20260925 03:05:03.571741 24472 rpcz_store.cc:276] 0925 03:05:01.650894 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:05:02.578802 (+927908us) service_pool.cc:227] Handling call
0925 03:05:02.579284 (+ 482us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:05:02.604902 (+ 25618us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:05:02.604947 (+ 45us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:05:02.605723 (+ 776us) log.cc:842] Serialized 26 byte log entry
0925 03:05:02.910999 (+305276us) raft_consensus.cc:1631] Marking committed up to 2
0925 03:05:02.911042 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:05:02.911119 (+ 77us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:05:03.539200 (+628081us) raft_consensus.cc:1671] finished
0925 03:05:03.539226 (+ 26us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:05:03.539371 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":217066,"raft.run_cpu_time_us":345,"raft.run_wall_time_us":82027,"thread_start_us":166348,"threads_started":2,"wal-append.queue_time_us":400910}
I20260925 03:05:03.575309 27145 consensus_queue.cc:237] T 442d4a29b1d743599f4b6b4871e26668 P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:05:03.575085 24540 rpcz_store.cc:269] 0925 03:04:58.791588 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:59.134976 (+343388us) service_pool.cc:227] Handling call
0925 03:05:03.569668 (+4434692us) spinlock_profiling.cc:243] Waited 601 ms on lock 0x7b540034b2d4. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d42ae 00007f437b411c27 00007f4374af9c04 00007f4374af9b92 00007f4374af9ac1 00007f4374af9a3c 00007f4374af8cf2 00007f43748fa94c 00007f43748f9d86 00007f43748f972f
0925 03:05:03.573407 (+ 3739us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1382981504}
I20260925 03:05:03.576247 26955 leader_election.cc:288] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:03.577649 26900 leader_election.cc:288] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:05:03.581627 26985 leader_election.cc:288] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
W20260925 03:05:03.583914 27150 log.cc:924] Time spent T c88cbd39e01f44628a094af516be2833 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.224s user 0.001s sys 0.000s
I20260925 03:05:03.586756 26980 leader_election.cc:288] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 8 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:05:03.589148 24545 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 1526) took 1682 ms. Trace:
I20260925 03:05:03.590598 27107 leader_election.cc:288] T a7aa47c48404465b94a36fd819fe3c7a P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:03.549539 26800 leader_election.cc:288] T c85f5d1d3f004cf5817bb594fd8e6469 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:03.591979 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc2df40db08649c9a75e90000b878559" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:03.593264 24616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9357d144aa49789faaf31fe5e6bf89" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:03.593506 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "420163a6cdce412ebc21c25fe3835638" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:03.594273 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1414dcfe53f64a78ae413531eb8ec117" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
W20260925 03:05:03.594530 25162 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 44671.4us
I20260925 03:05:03.589743 24545 rpcz_store.cc:276] 0925 03:05:01.907189 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:05:01.907312 (+ 123us) service_pool.cc:227] Handling call
0925 03:05:01.907622 (+ 310us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:05:02.966216 (+1058594us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:05:02.972935 (+ 6719us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:05:03.084829 (+111894us) log.cc:842] Serialized 26 byte log entry
0925 03:05:03.263584 (+178755us) raft_consensus.cc:1631] Marking committed up to 3
0925 03:05:03.263620 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:05:03.270497 (+ 6877us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:05:03.588762 (+318265us) raft_consensus.cc:1671] finished
0925 03:05:03.588787 (+ 25us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:05:03.589051 (+ 264us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":379096,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":135,"thread_start_us":1036798,"threads_started":2,"wal-append.queue_time_us":225868}
I20260925 03:05:03.588999 27083 leader_election.cc:288] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:03.597146 27105 raft_consensus.cc:2781] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260925 03:05:03.602784 24514 leader_election.cc:334] T 3d89396576884f7c8901dba54f783947 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187): Timed out: RequestConsensusVote RPC to 127.22.193.1:42187 timed out after 1.635s (SENT)
I20260925 03:05:03.597077 27135 leader_election.cc:288] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:03.604558 24548 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 1190) took 3154 ms. Trace:
I20260925 03:05:03.592603 27124 leader_election.cc:288] T a7aa47c48404465b94a36fd819fe3c7a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:03.551731 27118 leader_election.cc:288] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:05:03.617280 24613 raft_consensus.cc:1226] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 6. Current term is 7. Ops: [4.4-6.5]
I20260925 03:05:03.613240 24514 leader_election.cc:302] T 3d89396576884f7c8901dba54f783947 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:03.605772 27105 raft_consensus.cc:482] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:05:03.593300 27101 leader_election.cc:288] T 5278a60f97da4e10aecd50a8d9a9a43c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:03.634724 25194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367277daf8d04778b3fa4ed94d71580d" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
W20260925 03:05:03.636950 25163 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 116114us
I20260925 03:05:03.637059 24619 raft_consensus.cc:1226] T 04e068a0f80941dc9138cee063f53bab P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 4. Current term is 5. Ops: []
I20260925 03:05:03.638028 25163 leader_election.cc:302] T 367277daf8d04778b3fa4ed94d71580d P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:03.639760 25163 leader_election.cc:302] T a4abe0ea757e4886be4945c5001e529a P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:03.645972 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9c33fd100ff44c1a3247319a324a044" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:05:03.646934 24619 raft_consensus.cc:2354] T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 6. Current term is 7.
I20260925 03:05:03.648746 27123 leader_election.cc:288] T 25102ba08e4e49388f6176badcd10a25 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:03.650192 24546 raft_consensus.cc:1261] T 0b5df303cd6a4d59aca5c18bbbff24ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260925 03:05:03.653941 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bb46bce3564841b7674e6a693bd2e6" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:03.609829 24548 rpcz_store.cc:276] 0925 03:05:00.316198 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:05:00.419061 (+102863us) service_pool.cc:227] Handling call
0925 03:05:00.419377 (+ 316us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:05:00.419605 (+ 228us) raft_consensus.cc:1520] Early marking committed up to index 3
0925 03:05:00.542798 (+123193us) log.cc:842] Serialized 14 byte log entry
0925 03:05:02.988507 (+2445709us) spinlock_profiling.cc:243] Waited 4.84 ms on lock 0x7b1800428140. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f437c4478b1 00007f437c454661 00007f437351f4ca 00007f437cc00f99 00007f43733e3990 00007f43735a5864 00007f43735a5155 00007f4379ea5167 00007f43764da807 00007f43764e9f0f 00007f43764e9e88
0925 03:05:03.025769 (+ 37262us) log.cc:842] Serialized 14 byte log entry
0925 03:05:03.363550 (+337781us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:05:03.372979 (+ 9429us) raft_consensus.cc:1631] Marking committed up to 3
0925 03:05:03.416932 (+ 43953us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:05:03.440633 (+ 23701us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:05:03.470277 (+ 29644us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2473220,"prepare.queue_time_us":2748325,"prepare.run_cpu_time_us":92,"prepare.run_wall_time_us":95,"raft.queue_time_us":31746,"raft.run_cpu_time_us":295,"raft.run_wall_time_us":13078,"spinlock_wait_cycles":11124608,"thread_start_us":232697,"threads_started":1,"wal-append.queue_time_us":300967}
I20260925 03:05:03.627930 27105 raft_consensus.cc:3037] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [term 9 FOLLOWER]: Advancing to term 10
I20260925 03:05:03.680147 24541 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33068 (request call id 164) took 1113 ms. Trace:
I20260925 03:05:03.686323 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "550118bc1a0641ce9803bdbf43f2069f" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:03.693346 24617 raft_consensus.cc:2396] T 550118bc1a0641ce9803bdbf43f2069f P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260925 03:05:03.725874 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ac466793d094325ace224708b2158ed" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:03.727252 27040 raft_consensus.cc:3032] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [term 7 LEADER]: Stepping down as leader of term 7
I20260925 03:05:03.727741 27040 raft_consensus.cc:729] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [term 7 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:05:03.734427 27019 leader_election.cc:288] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:05:03.606231 27142 raft_consensus.cc:504] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:05:03.743364 27056 log.cc:924] Time spent T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 3.528s user 0.001s sys 0.017s
I20260925 03:05:03.750291 24542 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33068 (request call id 168) took 1174 ms. Trace:
I20260925 03:05:03.750790 24542 rpcz_store.cc:276] 0925 03:05:02.576141 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:05:02.582333 (+ 6192us) service_pool.cc:227] Handling call
0925 03:05:03.750055 (+1167722us) spinlock_profiling.cc:243] Waited 761 ms on lock 0x7b54002819d4. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d42ae 00007f437b411c27 00007f4374af9c04 00007f4374af9b92 00007f4374af9ac1 00007f4374af9a3c 00007f4374af8cf2 00007f43748fa94c 00007f43748f9d86 00007f43748f972f
0925 03:05:03.750215 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1751376256}
I20260925 03:05:03.752628 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:05:03.682483 24541 rpcz_store.cc:276] 0925 03:05:02.567241 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:05:02.573183 (+ 5942us) service_pool.cc:227] Handling call
0925 03:05:03.679876 (+1106693us) spinlock_profiling.cc:243] Waited 439 ms on lock 0x7b540034b7d4. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d42ae 00007f437b411c27 00007f4374af9c04 00007f4374af9b92 00007f4374af9ac1 00007f4374af9a3c 00007f4374af8cf2 00007f43748fa94c 00007f43748f9d86 00007f43748f972f
0925 03:05:03.680046 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1010452864}
I20260925 03:05:03.677269 27118 raft_consensus.cc:2726] T 3d89396576884f7c8901dba54f783947 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260925 03:05:03.859109 24614 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34546 (request call id 172) took 4007 ms. Trace:
I20260925 03:05:03.860278 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60d45b7d467d4e1ab92367f6d7d52b15" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:03.859584 24614 rpcz_store.cc:276] 0925 03:04:59.834156 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:59.835409 (+ 1253us) service_pool.cc:227] Handling call
0925 03:04:59.835751 (+ 342us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:04:59.838917 (+ 3166us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:04:59.838951 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:04:59.839391 (+ 440us) log.cc:842] Serialized 26 byte log entry
0925 03:05:00.021826 (+182435us) raft_consensus.cc:1631] Marking committed up to 7
0925 03:05:00.021877 (+ 51us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:05:00.021946 (+ 69us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:05:03.742516 (+3720570us) spinlock_profiling.cc:243] Waited 3.14 ms on lock 0x7b44000232cc. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f43733ecf11 00007f43734baa95 00007f43734bb4b2 00007f4376163511 00007f437615ee1a 00007f4376152bee 00007f4376151f36 00007f4376151b58 00007f437615b551 00007f437615b509 00007f437615b499
0925 03:05:03.758581 (+ 16065us) raft_consensus.cc:1671] finished
0925 03:05:03.758605 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:05:03.841321 (+ 82716us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":16653,"raft.queue_time_us":124785,"raft.run_cpu_time_us":120,"raft.run_wall_time_us":5342,"spinlock_wait_cycles":7211776,"thread_start_us":182136,"threads_started":1,"wal-append.queue_time_us":334253}
I20260925 03:05:03.862293 24548 raft_consensus.cc:2379] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c in current term 8: Already voted for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in this term.
I20260925 03:05:03.859699 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e068a0f80941dc9138cee063f53bab" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:05:03.864253 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "550118bc1a0641ce9803bdbf43f2069f" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:03.865056 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f68804eb5a244d87bb395d50683c6012" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:05:03.852551 27125 leader_election.cc:288] T c85f5d1d3f004cf5817bb594fd8e6469 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:03.712591 26923 raft_consensus.cc:2726] T a4abe0ea757e4886be4945c5001e529a P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260925 03:05:03.753720 25191 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49794 (request call id 269) took 1342 ms. Trace:
I20260925 03:05:03.852366 27147 leader_election.cc:288] T c88cbd39e01f44628a094af516be2833 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:05:03.871829 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5332a5d329493991dff994e70ea92c" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:03.873634 26861 leader_election.cc:288] T b4ebd2b043024a548005c8b6b00b0449 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:05:03.863497 27143 raft_consensus.cc:2726] T 367277daf8d04778b3fa4ed94d71580d P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260925 03:05:03.874469 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367277daf8d04778b3fa4ed94d71580d" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
W20260925 03:05:03.877297 24511 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 304759us
W20260925 03:05:03.868670 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 315252us
I20260925 03:05:03.880406 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35992316cb8d4c028333388eff02ecde" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:03.880466 24614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:03.883212 24614 raft_consensus.cc:2379] T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 8: Already voted for candidate eeab93abfdb141a895002ec9f312296c in this term.
I20260925 03:05:03.881983 24541 raft_consensus.cc:2454] T 550118bc1a0641ce9803bdbf43f2069f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 6.
I20260925 03:05:03.880636 24615 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38570 (request call id 1588) took 1890 ms. Trace:
I20260925 03:05:03.885119 24546 raft_consensus.cc:1226] T 21c43af5e6844fdaacc8c625fa01fd8d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 4. Current term is 5. Ops: []
I20260925 03:05:03.884882 24615 rpcz_store.cc:276] 0925 03:05:01.986652 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:05:01.987090 (+ 438us) service_pool.cc:227] Handling call
0925 03:05:03.876215 (+1889125us) spinlock_profiling.cc:243] Waited 3.33 ms on lock 0x7b58001d3d98. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f43733ecf11 00007f4373570851 00007f4373570702 00007f437c460b09 00007f437b39c730 00007f437b46279b 00007f437b462821 00007f437b4262b0 00007f437b4117f9 00007f437b4112a2 00007f4374af7684
0925 03:05:03.876462 (+ 247us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:05:03.876671 (+ 209us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:05:03.876711 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:05:03.876762 (+ 51us) raft_consensus.cc:1631] Marking committed up to 4
0925 03:05:03.876788 (+ 26us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:05:03.876840 (+ 52us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:05:03.876927 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7667456}
I20260925 03:05:03.893147 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e5819c15eac4544a57e5ee6c9d4f117" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:03.895522 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b8e0d260a884ecdb20f79de46c36b19" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:03.881616 27142 leader_election.cc:288] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:03.832475 27040 consensus_queue.cc:260] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
W20260925 03:05:03.903585 24514 leader_election.cc:334] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187): Timed out: RequestConsensusVote RPC to 127.22.193.1:42187 timed out after 1.755s (SENT)
I20260925 03:05:03.869010 25191 rpcz_store.cc:276] 0925 03:05:02.411004 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:05:02.411211 (+ 207us) service_pool.cc:227] Handling call
0925 03:05:02.411615 (+ 404us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:05:03.380231 (+968616us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:05:03.753373 (+373142us) spinlock_profiling.cc:243] Waited 899 ms on lock 0x7b54005de8d4. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d0f51 00007f43764cf99e 00007f437b4112e9 00007f4374af7684 00007f4374af7612 00007f4374af7541 00007f4374af74bc 00007f4374af6772 00007f43748fa94c 00007f43748f9d86
0925 03:05:03.753628 (+ 255us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2066984832}
W20260925 03:05:03.915984 25163 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 276324us
I20260925 03:05:03.920424 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d89396576884f7c8901dba54f783947" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:05:03.925752 24546 raft_consensus.cc:3037] T 3d89396576884f7c8901dba54f783947 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:05:03.930534 24615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43366a0a2f9d4344bf4e37ffd290ef8a" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:03.943945 25163 leader_election.cc:302] T 04e068a0f80941dc9138cee063f53bab P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
W20260925 03:05:03.947670 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 47041.6us
I20260925 03:05:03.949839 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1414dcfe53f64a78ae413531eb8ec117" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
W20260925 03:05:03.956236 24514 leader_election.cc:334] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187): Timed out: RequestConsensusVote RPC to 127.22.193.1:42187 timed out after 1.795s (SENT)
I20260925 03:05:03.956811 24514 leader_election.cc:302] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:03.957649 27118 raft_consensus.cc:2726] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260925 03:05:03.958796 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74443a0a1dc34d88a2f42e3b722db405" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:03.960273 25191 raft_consensus.cc:1226] T a4abe0ea757e4886be4945c5001e529a P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 2. Current term is 3. Ops: []
I20260925 03:05:03.961180 27063 leader_election.cc:288] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:03.962347 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35992316cb8d4c028333388eff02ecde" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:05:03.962980 24539 raft_consensus.cc:3037] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:05:03.962868 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06f4a7eaf6534e4986c9fd35911038e4" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:03.964083 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37a157996e944db2906110d87ecb0c54" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:03.964633 24540 raft_consensus.cc:2454] T 37a157996e944db2906110d87ecb0c54 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 7.
I20260925 03:05:03.933183 27105 raft_consensus.cc:504] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:03.974906 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7691c6b475428c82d6181d693d26f3" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:03.972265 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71f461a6e2ff434993b78a2d077f0abd" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:03.976759 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ac285175464fbda67211d9574db415" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:03.978267 24540 raft_consensus.cc:2454] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 6.
I20260925 03:05:03.978816 25199 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48424 (request call id 244) took 3614 ms. Trace:
I20260925 03:05:03.979040 24466 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1271) took 2327 ms. Trace:
I20260925 03:05:03.980068 24466 rpcz_store.cc:276] 0925 03:05:01.651389 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:05:02.590472 (+939083us) service_pool.cc:227] Handling call
0925 03:05:02.602292 (+ 11820us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:05:03.223049 (+620757us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:05:03.223080 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:05:03.224218 (+ 1138us) raft_consensus.cc:1631] Marking committed up to 2
0925 03:05:03.224251 (+ 33us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:05:03.978794 (+754543us) spinlock_profiling.cc:243] Waited 603 ms on lock 0x7b54003106d4. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d0f51 00007f43764cf99e 00007f437b4112e9 00007f4374af7684 00007f4374af7612 00007f4374af7541 00007f4374af74bc 00007f4374af6772 00007f43748fa94c 00007f43748f9d86
0925 03:05:03.978856 (+ 62us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:05:03.978985 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1387973888}
I20260925 03:05:03.979249 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceab6b0fccd54e909a64f66fe81a9a1f" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:05:03.982007 24540 raft_consensus.cc:3037] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:05:03.982422 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b5df303cd6a4d59aca5c18bbbff24ed" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:03.988183 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21c43af5e6844fdaacc8c625fa01fd8d" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
W20260925 03:05:03.989746 25163 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 67760.6us
I20260925 03:05:03.991547 25163 leader_election.cc:302] T e6b6ec010caf4689b8c90449764f5cf9 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:05:03.994832 24540 raft_consensus.cc:2454] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 7.
I20260925 03:05:03.995860 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:03.996613 24540 raft_consensus.cc:2379] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 in current term 8: Already voted for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in this term.
I20260925 03:05:04.019209 27105 leader_election.cc:288] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 10 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:04.019937 27105 raft_consensus.cc:3037] T e6b6ec010caf4689b8c90449764f5cf9 P 1bc1f2e62ab44af995165bd5116a9d43 [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:05:04.020407 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e54ca1757ee4e42956a9cebbeca4f62" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:05:04.021121 24617 raft_consensus.cc:3037] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Advancing to term 10
I20260925 03:05:04.021780 27134 leader_election.cc:288] T e6b6ec010caf4689b8c90449764f5cf9 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:05:04.029775 24617 raft_consensus.cc:2454] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 10.
I20260925 03:05:04.031247 25162 leader_election.cc:302] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:05:03.977208 27146 consensus_queue.cc:1046] T a4abe0ea757e4886be4945c5001e529a P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.056s
I20260925 03:05:04.033960 27071 raft_consensus.cc:3032] T a4abe0ea757e4886be4945c5001e529a P eeab93abfdb141a895002ec9f312296c [term 2 LEADER]: Stepping down as leader of term 2
I20260925 03:05:04.034382 27071 raft_consensus.cc:729] T a4abe0ea757e4886be4945c5001e529a P eeab93abfdb141a895002ec9f312296c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:05:04.035425 27071 consensus_queue.cc:260] T a4abe0ea757e4886be4945c5001e529a P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:04.037595 27071 raft_consensus.cc:3037] T a4abe0ea757e4886be4945c5001e529a P eeab93abfdb141a895002ec9f312296c [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:05:03.984252 24542 raft_consensus.cc:2454] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 7.
I20260925 03:05:03.972479 27167 raft_consensus.cc:2726] T 04e068a0f80941dc9138cee063f53bab P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260925 03:05:04.051728 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74443a0a1dc34d88a2f42e3b722db405" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:04.052539 24542 raft_consensus.cc:2454] T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 5.
W20260925 03:05:04.055364 24514 leader_election.cc:334] T b4ebd2b043024a548005c8b6b00b0449 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187): Timed out: RequestConsensusVote RPC to 127.22.193.1:42187 timed out after 1.839s (SENT)
I20260925 03:05:04.055925 24514 leader_election.cc:302] T b4ebd2b043024a548005c8b6b00b0449 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:05:04.057016 24546 raft_consensus.cc:2454] T 3d89396576884f7c8901dba54f783947 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 5.
I20260925 03:05:04.058077 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7993f9c3e2a44f408bbf5d1e395e2306" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:04.045758 27051 raft_consensus.cc:2781] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [term 10 FOLLOWER]: Leader election won for term 10
I20260925 03:05:04.058794 24546 raft_consensus.cc:2454] T 7993f9c3e2a44f408bbf5d1e395e2306 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 5.
I20260925 03:05:03.979202 25199 rpcz_store.cc:276] 0925 03:05:00.364079 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:05:00.364219 (+ 140us) service_pool.cc:227] Handling call
0925 03:05:00.364529 (+ 310us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:05:00.364756 (+ 227us) raft_consensus.cc:1520] Early marking committed up to index 8
0925 03:05:00.552312 (+187556us) log.cc:842] Serialized 14 byte log entry
0925 03:05:03.962117 (+3409805us) spinlock_profiling.cc:243] Waited 5.93 ms on lock 0x7b18002c6320. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f437c4478b1 00007f437c454661 00007f437615f82c 00007f437358a9dc 00007f437c3c1a39 00007f43735a78bb 00007f43735a5cbf 00007f43735a7e22 00007f43761515bb 00007f4376157063 00007f437615757b
0925 03:05:03.978187 (+ 16070us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:05:03.978291 (+ 104us) raft_consensus.cc:1631] Marking committed up to 8
0925 03:05:03.978381 (+ 90us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:05:03.978491 (+ 110us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:05:03.978637 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":17449,"prepare.queue_time_us":335593,"prepare.run_cpu_time_us":60,"prepare.run_wall_time_us":4253,"raft.queue_time_us":13853,"spinlock_wait_cycles":13642496,"thread_start_us":372041,"threads_started":2,"wal-append.queue_time_us":522687,"wal-append.run_cpu_time_us":1756,"wal-append.run_wall_time_us":1004253}
I20260925 03:05:04.081310 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9357d144aa49789faaf31fe5e6bf89" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:04.085173 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b8e0d260a884ecdb20f79de46c36b19" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:04.085906 24544 raft_consensus.cc:2454] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 6.
I20260925 03:05:04.086095 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c88cbd39e01f44628a094af516be2833" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:04.086774 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1414dcfe53f64a78ae413531eb8ec117" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:04.087570 24544 raft_consensus.cc:2396] T 1414dcfe53f64a78ae413531eb8ec117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260925 03:05:04.088068 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d89396576884f7c8901dba54f783947" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:04.088286 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ac285175464fbda67211d9574db415" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:04.090637 24539 raft_consensus.cc:2454] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 7.
I20260925 03:05:04.089155 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e92f377a79354a6ebb4eec67303c13e3" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:04.097942 24615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7993f9c3e2a44f408bbf5d1e395e2306" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:04.100800 24544 raft_consensus.cc:2454] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 6.
I20260925 03:05:04.100423 24546 raft_consensus.cc:2354] T 3d89396576884f7c8901dba54f783947 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 4. Current term is 5.
I20260925 03:05:04.104152 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37a157996e944db2906110d87ecb0c54" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:04.109880 26800 raft_consensus.cc:2726] T b4ebd2b043024a548005c8b6b00b0449 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260925 03:05:04.110530 24615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48ca6ccd98194cf2af7b5278d15bf4ea" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:04.110817 26811 leader_election.cc:288] T edcf6e6a422d4d198d4fbb0f7d3d845d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:04.110009 25192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71f461a6e2ff434993b78a2d077f0abd" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:05:04.111332 26937 raft_consensus.cc:482] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:05:04.111940 26937 raft_consensus.cc:504] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:04.112494 24615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f1fac028a144058b92331c0d9b44ebf" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
W20260925 03:05:04.113879 25192 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48424 (request call id 246) took 3272 ms (client timeout 2105 ms). Trace:
W20260925 03:05:04.114322 25192 rpcz_store.cc:269] 0925 03:05:00.841118 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:05:00.844141 (+ 3023us) service_pool.cc:227] Handling call
0925 03:05:04.081019 (+3236878us) spinlock_profiling.cc:243] Waited 19.1 ms on lock 0x7b5800004e98. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f43733ecf11 00007f4373570851 00007f4373570702 00007f437c460b09 00007f437b39c730 00007f437b46279b 00007f437b462821 00007f437b4121b0 00007f437b411a94 00007f4374af9c04 00007f4374af9b92
0925 03:05:04.113805 (+ 32786us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":44002176}
I20260925 03:05:04.101645 27051 raft_consensus.cc:686] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [term 10 LEADER]: Becoming Leader. State: Replica: 1bc1f2e62ab44af995165bd5116a9d43, State: Running, Role: LEADER
I20260925 03:05:04.117007 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:05:04.117197 24615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b030d84c670b48d5bff337ccc0c77626" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:04.117875 24620 raft_consensus.cc:2379] T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 8: Already voted for candidate eeab93abfdb141a895002ec9f312296c in this term.
W20260925 03:05:04.118158 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 75180.1us
I20260925 03:05:04.118402 24615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7aa47c48404465b94a36fd819fe3c7a" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:04.118767 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c23337a7cb74184a0e57b7095cb6a11" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
W20260925 03:05:04.119388 24514 leader_election.cc:334] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187): Timed out: RequestConsensusVote RPC to 127.22.193.1:42187 timed out after 1.933s (SENT)
I20260925 03:05:04.121388 24544 raft_consensus.cc:1261] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260925 03:05:04.125108 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c85f5d1d3f004cf5817bb594fd8e6469" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
W20260925 03:05:04.113291 25163 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 62982.2us
I20260925 03:05:04.103550 27126 leader_election.cc:288] T c88cbd39e01f44628a094af516be2833 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:04.141554 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be4e0aa984f4f2b81a58e382e434ad5" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:04.142935 25163 leader_election.cc:302] T 71f461a6e2ff434993b78a2d077f0abd P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:05:04.145771 24542 raft_consensus.cc:1261] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20260925 03:05:04.147662 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5278a60f97da4e10aecd50a8d9a9a43c" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:04.162681 27009 consensus_queue.cc:1035] T 0b5df303cd6a4d59aca5c18bbbff24ed P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.016s
I20260925 03:05:04.148882 24586 leader_election.cc:302] T 04e068a0f80941dc9138cee063f53bab P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 1bc1f2e62ab44af995165bd5116a9d43, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:05:04.173110 27051 consensus_queue.cc:237] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:04.173861 24547 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 1177) took 3864 ms. Trace:
I20260925 03:05:04.174206 24547 rpcz_store.cc:276] 0925 03:05:00.309728 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:05:00.313367 (+ 3639us) service_pool.cc:227] Handling call
0925 03:05:00.313695 (+ 328us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:05:01.084780 (+771085us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:05:04.173483 (+3088703us) spinlock_profiling.cc:243] Waited 608 ms on lock 0x7b540010de54. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d0f51 00007f43764cf99e 00007f437b4112e9 00007f4374af7684 00007f4374af7612 00007f4374af7541 00007f4374af74bc 00007f4374af6772 00007f43748fa94c 00007f43748f9d86
0925 03:05:04.173713 (+ 230us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1398862592}
I20260925 03:05:04.178277 24539 raft_consensus.cc:1261] T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260925 03:05:04.180258 24539 raft_consensus.cc:1226] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 6. Current term is 7. Ops: []
I20260925 03:05:04.183099 26820 raft_consensus.cc:2781] T 71f461a6e2ff434993b78a2d077f0abd P 1bc1f2e62ab44af995165bd5116a9d43 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260925 03:05:04.183581 26820 raft_consensus.cc:482] T 71f461a6e2ff434993b78a2d077f0abd P 1bc1f2e62ab44af995165bd5116a9d43 [term 7 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:04.183979 26820 raft_consensus.cc:3037] T 71f461a6e2ff434993b78a2d077f0abd P 1bc1f2e62ab44af995165bd5116a9d43 [term 7 FOLLOWER]: Advancing to term 8
I20260925 03:05:04.184357 24615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e725b82985746b39c0cc8f671c95353" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:04.188621 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25102ba08e4e49388f6176badcd10a25" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:04.193737 24619 raft_consensus.cc:1261] T a4abe0ea757e4886be4945c5001e529a P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260925 03:05:04.167856 26800 raft_consensus.cc:482] T 442d4a29b1d743599f4b6b4871e26668 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:04.195976 24615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceab6b0fccd54e909a64f66fe81a9a1f" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:04.192909 26811 raft_consensus.cc:482] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:04.201735 26811 raft_consensus.cc:504] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:04.203869 26811 leader_election.cc:288] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
W20260925 03:05:04.205343 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 57346us
I20260925 03:05:04.207664 24586 leader_election.cc:302] T 550118bc1a0641ce9803bdbf43f2069f P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:05:04.208909 27009 consensus_queue.cc:1046] T 21c43af5e6844fdaacc8c625fa01fd8d P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260925 03:05:04.211428 27009 raft_consensus.cc:3032] T 21c43af5e6844fdaacc8c625fa01fd8d P eeab93abfdb141a895002ec9f312296c [term 4 LEADER]: Stepping down as leader of term 4
I20260925 03:05:04.211758 27009 raft_consensus.cc:729] T 21c43af5e6844fdaacc8c625fa01fd8d P eeab93abfdb141a895002ec9f312296c [term 4 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:05:04.213243 26842 raft_consensus.cc:2781] T 550118bc1a0641ce9803bdbf43f2069f P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260925 03:05:04.213706 26842 raft_consensus.cc:482] T 550118bc1a0641ce9803bdbf43f2069f P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Starting leader election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:05:04.214052 26842 raft_consensus.cc:3037] T 550118bc1a0641ce9803bdbf43f2069f P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:05:04.214427 26800 raft_consensus.cc:504] T 442d4a29b1d743599f4b6b4871e26668 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:04.198439 27064 raft_consensus.cc:2726] T 04e068a0f80941dc9138cee063f53bab P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260925 03:05:04.197664 25163 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 62370.4us
I20260925 03:05:04.223801 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5f9b503b0ab43999d582015a367441e" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:04.216533 26800 leader_election.cc:288] T 442d4a29b1d743599f4b6b4871e26668 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:04.212560 27009 consensus_queue.cc:260] T 21c43af5e6844fdaacc8c625fa01fd8d P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:05:04.225628 24511 leader_election.cc:334] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105): Timed out: RequestConsensusVote RPC to 127.22.193.3:35105 timed out after 1.758s (SENT)
I20260925 03:05:04.223318 24614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4abe0ea757e4886be4945c5001e529a" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:05:04.235559 24615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "420163a6cdce412ebc21c25fe3835638" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:04.237141 24620 raft_consensus.cc:2454] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 8.
I20260925 03:05:04.224500 26820 raft_consensus.cc:504] T 71f461a6e2ff434993b78a2d077f0abd P 1bc1f2e62ab44af995165bd5116a9d43 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:04.242504 24615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "550118bc1a0641ce9803bdbf43f2069f" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
W20260925 03:05:04.253047 24618 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38570 (request call id 1589) took 2266 ms (client timeout 2214 ms). Trace:
W20260925 03:05:04.253525 24618 rpcz_store.cc:269] 0925 03:05:01.986919 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:05:01.988408 (+ 1489us) service_pool.cc:227] Handling call
0925 03:05:04.246758 (+2258350us) spinlock_profiling.cc:243] Waited 446 ms on lock 0x7b54002189d4. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d42ae 00007f437b411c27 00007f4374af9c04 00007f4374af9b92 00007f4374af9ac1 00007f4374af9a3c 00007f4374af8cf2 00007f43748fa94c 00007f43748f9d86 00007f43748f972f
0925 03:05:04.252938 (+ 6180us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1025763072}
I20260925 03:05:04.254736 24618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71f461a6e2ff434993b78a2d077f0abd" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:04.255967 24618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9c33fd100ff44c1a3247319a324a044" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:05:04.257905 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bb46bce3564841b7674e6a693bd2e6" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:04.260535 24618 raft_consensus.cc:2354] T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 6. Current term is 7.
I20260925 03:05:04.277124 27105 raft_consensus.cc:2726] T e6b6ec010caf4689b8c90449764f5cf9 P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260925 03:05:04.278371 27009 raft_consensus.cc:3037] T 21c43af5e6844fdaacc8c625fa01fd8d P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:05:04.279402 24545 raft_consensus.cc:1226] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 6. Current term is 7. Ops: []
I20260925 03:05:04.281872 25163 leader_election.cc:398] T 7e49248643df4131a1c42e74244e05c2 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 7 pre-election: Vote denied by peer a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389) with higher term. Message: Invalid argument: T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1bc1f2e62ab44af995165bd5116a9d43 for earlier term 7. Current term is 8.
I20260925 03:05:04.282251 25163 leader_election.cc:401] T 7e49248643df4131a1c42e74244e05c2 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 7 pre-election: Cancelling election due to peer responding with higher term
I20260925 03:05:04.284219 24467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f68804eb5a244d87bb395d50683c6012" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:04.284641 25198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1e85248d3b04b979dd080caeeb02fdb" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:05:04.285372 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcf6e6a422d4d198d4fbb0f7d3d845d" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:04.286540 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c88cbd39e01f44628a094af516be2833" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:04.286644 27094 leader_election.cc:288] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:04.287092 24617 raft_consensus.cc:2454] T c88cbd39e01f44628a094af516be2833 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 7.
I20260925 03:05:04.278236 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b627ecb4a5384ef392aa695e06599c39" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:04.294436 24615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9357d144aa49789faaf31fe5e6bf89" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:04.296495 25199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15861cd28e354598a3c6bfc4337c34bb" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43"
W20260925 03:05:04.299808 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 54278.3us
I20260925 03:05:04.301066 25199 raft_consensus.cc:2379] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 in current term 6: Already voted for candidate 1bc1f2e62ab44af995165bd5116a9d43 in this term.
I20260925 03:05:04.301551 24614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc2df40db08649c9a75e90000b878559" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:04.302753 24618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ac466793d094325ace224708b2158ed" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:04.304337 24586 leader_election.cc:302] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:05:04.304000 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b627ecb4a5384ef392aa695e06599c39" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:04.309165 25198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367277daf8d04778b3fa4ed94d71580d" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:05:04.310518 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442d4a29b1d743599f4b6b4871e26668" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:04.314481 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abd5b8dc1b7e49e3aa9af900d73ff0e7" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:04.315004 26820 leader_election.cc:288] T 71f461a6e2ff434993b78a2d077f0abd P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 8 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:04.315677 26803 raft_consensus.cc:2726] T 7e49248643df4131a1c42e74244e05c2 P 1bc1f2e62ab44af995165bd5116a9d43 [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: Vote denied by peer a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389) with higher term. Message: Invalid argument: T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1bc1f2e62ab44af995165bd5116a9d43 for earlier term 7. Current term is 8.
I20260925 03:05:04.317368 24467 raft_consensus.cc:3037] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:05:04.318408 26842 raft_consensus.cc:504] T 550118bc1a0641ce9803bdbf43f2069f P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:04.321564 24586 leader_election.cc:302] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:05:04.322759 26900 raft_consensus.cc:2781] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260925 03:05:04.322925 24586 leader_election.cc:302] T 3d89396576884f7c8901dba54f783947 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:05:04.323261 26900 raft_consensus.cc:482] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Starting leader election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:05:04.323724 26900 raft_consensus.cc:3037] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Advancing to term 8
I20260925 03:05:04.323866 27038 raft_consensus.cc:2781] T 3d89396576884f7c8901dba54f783947 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Leader election won for term 5
I20260925 03:05:04.324538 27038 raft_consensus.cc:686] T 3d89396576884f7c8901dba54f783947 P eeab93abfdb141a895002ec9f312296c [term 5 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:05:04.325616 27038 consensus_queue.cc:237] T 3d89396576884f7c8901dba54f783947 P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:04.328526 26756 consensus_queue.cc:1035] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260925 03:05:04.328846 26800 consensus_queue.cc:1035] T a4abe0ea757e4886be4945c5001e529a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260925 03:05:04.333750 25165 leader_election.cc:302] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:05:04.336790 25199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e92f377a79354a6ebb4eec67303c13e3" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:05:04.296263 27116 leader_election.cc:288] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:04.339355 24621 raft_consensus.cc:1261] T c88cbd39e01f44628a094af516be2833 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260925 03:05:04.312657 27064 raft_consensus.cc:2781] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260925 03:05:04.344065 24467 raft_consensus.cc:2454] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 7.
I20260925 03:05:04.351382 24618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bb46bce3564841b7674e6a693bd2e6" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:04.352326 25198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:05:04.312546 25192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15861cd28e354598a3c6bfc4337c34bb" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43"
I20260925 03:05:04.351164 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e92f377a79354a6ebb4eec67303c13e3" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:04.302057 27105 consensus_queue.cc:1035] T 8ac466793d094325ace224708b2158ed P 1bc1f2e62ab44af995165bd5116a9d43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.047s
I20260925 03:05:04.353044 25198 raft_consensus.cc:2454] T 29637f5a4a504ce6ba29033ccd0cd716 P 1bc1f2e62ab44af995165bd5116a9d43 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 8.
I20260925 03:05:04.358464 24467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "550118bc1a0641ce9803bdbf43f2069f" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:04.350857 27051 raft_consensus.cc:2781] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260925 03:05:04.360116 27051 raft_consensus.cc:482] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43 [term 9 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:04.364527 25199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e54ca1757ee4e42956a9cebbeca4f62" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:05:04.365281 24618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b627ecb4a5384ef392aa695e06599c39" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:04.365792 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48ca6ccd98194cf2af7b5278d15bf4ea" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:05:04.367677 24436 leader_election.cc:302] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: 1bc1f2e62ab44af995165bd5116a9d43, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:04.370747 24436 leader_election.cc:302] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:05:04.371611 27094 raft_consensus.cc:2726] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260925 03:05:04.351953 27064 raft_consensus.cc:482] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:05:04.377938 24465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d89396576884f7c8901dba54f783947" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:04.380072 24469 raft_consensus.cc:1226] T ceab6b0fccd54e909a64f66fe81a9a1f P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 6. Current term is 7. Ops: []
I20260925 03:05:04.380224 24465 raft_consensus.cc:3037] T 3d89396576884f7c8901dba54f783947 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Advancing to term 5
I20260925 03:05:04.381381 24618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "100c113a9fd84cc8adf2ec7b33ea0093" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:04.363291 27051 raft_consensus.cc:3037] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43 [term 9 FOLLOWER]: Advancing to term 10
I20260925 03:05:04.346477 26970 consensus_queue.cc:1035] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.029s
I20260925 03:05:04.385762 24470 raft_consensus.cc:1261] T 8e5819c15eac4544a57e5ee6c9d4f117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260925 03:05:04.386898 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be4e0aa984f4f2b81a58e382e434ad5" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:04.372743 25195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4ebd2b043024a548005c8b6b00b0449" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:05:04.385111 24473 raft_consensus.cc:1226] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 6. Current term is 7. Ops: []
I20260925 03:05:04.389716 24618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f1fac028a144058b92331c0d9b44ebf" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:04.384869 25192 raft_consensus.cc:2354] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 5. Current term is 6.
I20260925 03:05:04.402150 25199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43"
I20260925 03:05:04.380993 27064 raft_consensus.cc:3037] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:05:04.408514 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c88cbd39e01f44628a094af516be2833" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:04.407171 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7691c6b475428c82d6181d693d26f3" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:04.385568 26842 leader_election.cc:288] T 550118bc1a0641ce9803bdbf43f2069f P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 7 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:05:04.412938 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "420163a6cdce412ebc21c25fe3835638" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:04.399230 27094 raft_consensus.cc:2781] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260925 03:05:04.417418 24437 connection.cc:232] RPC call timeout handler was delayed by -1.4395641990013246s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2735s for a response
I20260925 03:05:04.422433 27094 raft_consensus.cc:482] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:05:04.422827 27094 raft_consensus.cc:3037] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:05:04.421840 24472 raft_consensus.cc:1261] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20260925 03:05:04.423084 26946 raft_consensus.cc:482] T 9e8af2641e124f9589c5ccfdac785f63 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:05:04.423738 26946 raft_consensus.cc:504] T 9e8af2641e124f9589c5ccfdac785f63 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:04.424345 24472 raft_consensus.cc:1226] T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 4. Current term is 5. Ops: []
I20260925 03:05:04.424847 27134 raft_consensus.cc:482] T 4e5332a5d329493991dff994e70ea92c P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:04.425407 27134 raft_consensus.cc:504] T 4e5332a5d329493991dff994e70ea92c P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:04.427315 27134 leader_election.cc:288] T 4e5332a5d329493991dff994e70ea92c P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:04.427898 27134 raft_consensus.cc:482] T 43366a0a2f9d4344bf4e37ffd290ef8a P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:04.429015 26509 consensus_queue.cc:1035] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.027s
I20260925 03:05:04.430001 26900 raft_consensus.cc:504] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:04.431088 24618 raft_consensus.cc:1226] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 7. Current term is 8. Ops: []
I20260925 03:05:04.429934 24614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60d45b7d467d4e1ab92367f6d7d52b15" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:04.434302 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "100c113a9fd84cc8adf2ec7b33ea0093" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:04.436293 24471 raft_consensus.cc:1261] T a7aa47c48404465b94a36fd819fe3c7a P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260925 03:05:04.438835 25198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71f461a6e2ff434993b78a2d077f0abd" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:05:04.427248 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5278a60f97da4e10aecd50a8d9a9a43c" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:04.442287 24468 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:55552 (request call id 449) took 5783 ms. Trace:
I20260925 03:05:04.442662 24468 rpcz_store.cc:276] 0925 03:04:58.658666 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:59.715854 (+1057188us) service_pool.cc:227] Handling call
0925 03:05:02.712501 (+2996647us) spinlock_profiling.cc:243] Waited 8.72 ms on lock 0x7b5800001598. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f43733ecf11 00007f4373570851 00007f4373570bce 000000000034aea9 00007f437b39ca30 00007f437b462741 00007f437b462900 00007f437b415b50 00007f4374b0c807 00007f4374b0c792 00007f4374b0c6c1
0925 03:05:04.441427 (+1728926us) spinlock_profiling.cc:243] Waited 321 ms on lock 0x7b5400141254. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455af0 00007f43764d8f2a 00007f437b415c6e 00007f4374b0c807 00007f4374b0c792 00007f4374b0c6c1 00007f4374b0c63c 00007f4374b0b8f2 00007f43748fa94c 00007f43748f9d86 00007f43748f972f
0925 03:05:04.442096 (+ 669us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":758628992}
I20260925 03:05:04.440449 27094 raft_consensus.cc:504] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:04.446748 25163 leader_election.cc:302] T e92f377a79354a6ebb4eec67303c13e3 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:04.447675 25199 raft_consensus.cc:2379] T 7e49248643df4131a1c42e74244e05c2 P 1bc1f2e62ab44af995165bd5116a9d43 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 8: Already voted for candidate eeab93abfdb141a895002ec9f312296c in this term.
I20260925 03:05:04.427343 24469 raft_consensus.cc:1261] T 4e5332a5d329493991dff994e70ea92c P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260925 03:05:04.450433 24471 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1263) took 2801 ms. Trace:
I20260925 03:05:04.452344 24471 rpcz_store.cc:276] 0925 03:05:01.649377 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:05:01.649491 (+ 114us) service_pool.cc:227] Handling call
0925 03:05:04.435633 (+2786142us) spinlock_profiling.cc:243] Waited 12.6 ms on lock 0x7b5800001598. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f43733ecf11 00007f4373570851 00007f4373570702 00007f437c460b09 00007f437b39c730 00007f437b46279b 00007f437b462821 00007f437b4262b0 00007f437b4117f9 00007f437b4112a2 00007f4374af7684
0925 03:05:04.435888 (+ 255us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:05:04.436649 (+ 761us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:05:04.450371 (+ 13722us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":28960128}
I20260925 03:05:04.459908 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ac285175464fbda67211d9574db415" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:04.460768 24473 raft_consensus.cc:2379] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c in current term 7: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:05:04.428328 27134 raft_consensus.cc:504] T 43366a0a2f9d4344bf4e37ffd290ef8a P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:04.407946 25191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6b6ec010caf4689b8c90449764f5cf9" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:05:04.470090 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7aa47c48404465b94a36fd819fe3c7a" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:04.463310 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71f461a6e2ff434993b78a2d077f0abd" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:05:04.470965 24541 raft_consensus.cc:3037] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Advancing to term 8
I20260925 03:05:04.424538 27104 leader_election.cc:288] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:04.426254 26946 leader_election.cc:288] T 9e8af2641e124f9589c5ccfdac785f63 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:04.417703 27107 raft_consensus.cc:482] T edcf6e6a422d4d198d4fbb0f7d3d845d P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:04.418335 27116 consensus_queue.cc:1046] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.054s
I20260925 03:05:04.480500 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abd5b8dc1b7e49e3aa9af900d73ff0e7" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:04.405082 26800 consensus_queue.cc:1035] T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.055s
I20260925 03:05:04.459647 27112 raft_consensus.cc:2726] T e92f377a79354a6ebb4eec67303c13e3 P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260925 03:05:04.483021 24436 leader_election.cc:398] T 3d89396576884f7c8901dba54f783947 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: Vote denied by peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105) with higher term. Message: Invalid argument: T 3d89396576884f7c8901dba54f783947 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 4. Current term is 5.
I20260925 03:05:04.486963 24436 leader_election.cc:401] T 3d89396576884f7c8901dba54f783947 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20260925 03:05:04.459451 26749 consensus_queue.cc:1046] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.021s
I20260925 03:05:04.470184 26811 consensus_queue.cc:1046] T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.018s
I20260925 03:05:04.480603 27126 raft_consensus.cc:482] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:05:04.492838 27126 raft_consensus.cc:504] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:05:04.482206 27173 log.cc:924] Time spent T 8ac466793d094325ace224708b2158ed P 1bc1f2e62ab44af995165bd5116a9d43: Append to log took a long time: real 0.098s user 0.000s sys 0.001s
I20260925 03:05:04.487830 27107 raft_consensus.cc:504] T edcf6e6a422d4d198d4fbb0f7d3d845d P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:05:04.496409 24588 connection.cc:232] RPC call timeout handler was delayed by -1.7039430719996744s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1626s for a response
W20260925 03:05:04.496840 24588 connection.cc:232] RPC call timeout handler was delayed by -1.6787430719996337s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1654s for a response
W20260925 03:05:04.497128 24588 connection.cc:232] RPC call timeout handler was delayed by -1.6346430720004719s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1703s for a response
W20260925 03:05:04.497457 24588 connection.cc:232] RPC call timeout handler was delayed by -1.6076430720004282s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1733s for a response
W20260925 03:05:04.497730 24588 connection.cc:232] RPC call timeout handler was delayed by -1.5644430720003584s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1781s for a response
W20260925 03:05:04.497982 24588 connection.cc:232] RPC call timeout handler was delayed by -1.1864430719997472s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22010000000000002s for a response
W20260925 03:05:04.498219 24588 connection.cc:232] RPC call timeout handler was delayed by -1.1783430720006436s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.221s for a response
W20260925 03:05:04.498457 24588 connection.cc:232] RPC call timeout handler was delayed by -1.1468430719996832s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22450000000000003s for a response
I20260925 03:05:04.499346 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c85f5d1d3f004cf5817bb594fd8e6469" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
W20260925 03:05:04.504923 24436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 47289.8us
I20260925 03:05:04.508503 26970 raft_consensus.cc:3032] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [term 6 LEADER]: Stepping down as leader of term 6
I20260925 03:05:04.495491 27058 raft_consensus.cc:3032] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 LEADER]: Stepping down as leader of term 6
I20260925 03:05:04.514566 26970 raft_consensus.cc:729] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [term 6 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:05:04.522861 27094 leader_election.cc:288] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:04.506568 27077 raft_consensus.cc:482] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:04.527737 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b030d84c670b48d5bff337ccc0c77626" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:04.509325 25191 raft_consensus.cc:2396] T e6b6ec010caf4689b8c90449764f5cf9 P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 for term 7 because replica has last-logged OpId of term: 5 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 7.
I20260925 03:05:04.563478 27077 raft_consensus.cc:504] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:04.565531 26756 consensus_queue.cc:1046] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260925 03:05:04.566221 24588 leader_election.cc:302] T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:05:04.588577 26842 raft_consensus.cc:3032] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [term 6 LEADER]: Stepping down as leader of term 6
I20260925 03:05:04.593807 26842 raft_consensus.cc:729] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [term 6 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:05:04.528523 27058 raft_consensus.cc:729] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:05:04.600055 27041 raft_consensus.cc:2726] T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260925 03:05:04.586813 27094 raft_consensus.cc:482] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:04.613407 27094 raft_consensus.cc:504] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:04.621204 26970 consensus_queue.cc:260] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:04.631314 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15861cd28e354598a3c6bfc4337c34bb" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:05:04.632076 24548 raft_consensus.cc:3037] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:05:04.616235 24436 leader_election.cc:302] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: 1bc1f2e62ab44af995165bd5116a9d43, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:04.662145 27058 consensus_queue.cc:260] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:05:04.663223 25196 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48408 (request call id 218) took 2726 ms (client timeout 1864 ms). Trace:
W20260925 03:05:04.663638 25196 rpcz_store.cc:269] 0925 03:05:01.936914 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:05:01.937125 (+ 211us) service_pool.cc:227] Handling call
0925 03:05:04.662985 (+2725860us) spinlock_profiling.cc:243] Waited 395 ms on lock 0x7b5400433ad4. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d42ae 00007f437b411c27 00007f4374af9c04 00007f4374af9b92 00007f4374af9ac1 00007f4374af9a3c 00007f4374af8cf2 00007f43748fa94c 00007f43748f9d86 00007f43748f972f
0925 03:05:04.663149 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":907942144}
I20260925 03:05:04.566708 26756 consensus_queue.cc:1046] T 5278a60f97da4e10aecd50a8d9a9a43c P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.066s
I20260925 03:05:04.694446 26577 raft_consensus.cc:3032] T 5278a60f97da4e10aecd50a8d9a9a43c P eeab93abfdb141a895002ec9f312296c [term 4 LEADER]: Stepping down as leader of term 4
I20260925 03:05:04.694924 26577 raft_consensus.cc:729] T 5278a60f97da4e10aecd50a8d9a9a43c P eeab93abfdb141a895002ec9f312296c [term 4 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:05:04.695874 26577 consensus_queue.cc:260] T 5278a60f97da4e10aecd50a8d9a9a43c P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:04.713479 27110 leader_election.cc:288] T b627ecb4a5384ef392aa695e06599c39 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:04.423892 23300 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260925 03:05:04.729377 24543 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 1519) took 4577 ms. Trace:
I20260925 03:05:04.729882 24543 rpcz_store.cc:276] 0925 03:05:00.145837 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:05:00.187521 (+ 41684us) service_pool.cc:227] Handling call
0925 03:05:00.216787 (+ 29266us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:05:00.217131 (+ 344us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:05:00.217160 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:05:03.894343 (+3677183us) spinlock_profiling.cc:243] Waited 1.73 ms on lock 0x7b2c00000238. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4374eb3e11 00007f4374ec3442 00007f437650bc11 00007f43764d058a 00007f43764cf99e 00007f437b4112e9 00007f4374af7684 00007f4374af7612 00007f4374af7541 00007f4374af74bc 00007f4374af6772
0925 03:05:03.894917 (+ 574us) log.cc:842] Serialized 26 byte log entry
0925 03:05:04.081576 (+186659us) raft_consensus.cc:1631] Marking committed up to 8
0925 03:05:04.110890 (+ 29314us) log.cc:842] Serialized 14 byte log entry
0925 03:05:04.111130 (+ 240us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:05:04.111209 (+ 79us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:05:04.720679 (+609470us) spinlock_profiling.cc:243] Waited 1.18 ms on lock 0x7b180033cca0. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f437c4478b1 00007f437c454661 00007f43735a8638 00007f43735abb11 00007f43735abac9 00007f43735aba59 00007f43735aba21 00007f43735aad1d 00007f437c80c204 00007f437c80c039 00007f437358adc6
0925 03:05:04.723378 (+ 2699us) raft_consensus.cc:1671] finished
0925 03:05:04.723406 (+ 28us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:05:04.723565 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":29724,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":39,"raft.queue_time_us":10215,"raft.run_cpu_time_us":270,"raft.run_wall_time_us":11992,"spinlock_wait_cycles":6688000,"thread_start_us":164263,"threads_started":1,"wal-append.queue_time_us":180593}
W20260925 03:05:04.732843 27158 log.cc:924] Time spent T 550118bc1a0641ce9803bdbf43f2069f P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.905s user 0.004s sys 0.002s
I20260925 03:05:04.594808 26842 consensus_queue.cc:260] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:04.741834 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:04.748277 26842 raft_consensus.cc:3037] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:05:04.752379 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1e85248d3b04b979dd080caeeb02fdb" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
W20260925 03:05:04.774067 27181 log.cc:924] Time spent T a7aa47c48404465b94a36fd819fe3c7a P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.213s user 0.001s sys 0.000s
I20260925 03:05:04.774811 24616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4abe0ea757e4886be4945c5001e529a" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:05:04.779486 26970 raft_consensus.cc:3037] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:05:04.799861 27064 raft_consensus.cc:504] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:04.826879 27064 leader_election.cc:288] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 7 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:05:04.802517 27134 leader_election.cc:288] T 43366a0a2f9d4344bf4e37ffd290ef8a P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:04.803370 26577 raft_consensus.cc:3037] T 5278a60f97da4e10aecd50a8d9a9a43c P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:05:04.806792 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e54ca1757ee4e42956a9cebbeca4f62" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:04.829068 24617 raft_consensus.cc:2379] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 in current term 10: Already voted for candidate 1bc1f2e62ab44af995165bd5116a9d43 in this term.
I20260925 03:05:04.830163 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d89396576884f7c8901dba54f783947" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:04.830391 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6b6ec010caf4689b8c90449764f5cf9" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:04.830709 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7691c6b475428c82d6181d693d26f3" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:04.831190 24466 raft_consensus.cc:2354] T e6b6ec010caf4689b8c90449764f5cf9 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 5. Current term is 6.
I20260925 03:05:04.831725 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1414dcfe53f64a78ae413531eb8ec117" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:04.810592 27077 leader_election.cc:288] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:05:04.812211 26937 leader_election.cc:288] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:05:04.809552 24371 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:57656 (request call id 84) took 9215 ms. Trace:
I20260925 03:05:04.833245 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9357d144aa49789faaf31fe5e6bf89" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:04.812862 27040 raft_consensus.cc:3037] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Advancing to term 8
I20260925 03:05:04.835474 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7691c6b475428c82d6181d693d26f3" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:05:04.836093 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc2df40db08649c9a75e90000b878559" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:04.814446 24436 leader_election.cc:302] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:05:04.817306 26900 leader_election.cc:288] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 8 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:05:04.817341 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "550118bc1a0641ce9803bdbf43f2069f" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:05:04.854171 24465 raft_consensus.cc:2454] T 3d89396576884f7c8901dba54f783947 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 5.
I20260925 03:05:04.819025 27058 raft_consensus.cc:3037] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:05:04.825119 24511 leader_election.cc:302] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:05:04.801821 27126 leader_election.cc:288] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:04.831900 26800 raft_consensus.cc:3032] T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 LEADER]: Stepping down as leader of term 4
I20260925 03:05:04.832496 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "420163a6cdce412ebc21c25fe3835638" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:04.833006 24371 rpcz_store.cc:276] 0925 03:04:55.593700 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:55.597341 (+ 3641us) service_pool.cc:227] Handling call
0925 03:05:04.809292 (+9211951us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0925 03:04:57.867760 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0925 03:04:57.868286 (+ 526us) write_op.cc:432] Acquiring schema lock in shared mode
0925 03:04:57.868330 (+ 44us) write_op.cc:435] Acquired schema lock
0925 03:04:57.868342 (+ 12us) tablet.cc:660] Decoding operations
0925 03:04:57.869040 (+ 698us) write_op.cc:620] Acquiring the partition lock for write op
0925 03:04:57.869097 (+ 57us) write_op.cc:641] Partition lock acquired for write op
0925 03:04:57.869128 (+ 31us) tablet.cc:683] Acquiring locks for 4 operations
0925 03:04:57.869503 (+ 375us) tablet.cc:699] Row locks acquired
0925 03:04:57.869526 (+ 23us) write_op.cc:260] PREPARE: finished
0925 03:04:57.869678 (+ 152us) write_op.cc:270] Start()
0925 03:04:57.869788 (+ 110us) write_op.cc:276] Timestamp: P: 1790305497869628 usec, L: 0
0925 03:04:57.869805 (+ 17us) op_driver.cc:331] REPLICATION: starting
0925 03:04:57.870454 (+ 649us) log.cc:842] Serialized 1797 byte log entry
0925 03:04:58.863878 (+993424us) spinlock_profiling.cc:243] Waited 48.1 ms on lock 0x7b44000232cc. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f43733ecf11 00007f43734baa95 00007f43734bb4b2 00007f4376163511 00007f437615ee1a 00007f4376152bee 00007f4376151f36 00007f4376151b58 00007f437615b551 00007f437615b509 00007f437615b499
0925 03:04:59.816669 (+952791us) op_driver.cc:444] REPLICATION: finished
0925 03:05:02.018797 (+2202128us) spinlock_profiling.cc:243] Waited 17.6 ms on lock 0x7b44000232cc. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f43733ecf11 00007f43734baa95 00007f43734bb4b2 00007f4376163511 00007f437615ee1a 00007f437358a5fe 00007f437c3c1a39 00007f43735a78bb 00007f43735a5cbf 00007f43735a7e22 00007f4379ee0410
0925 03:05:02.368014 (+349217us) write_op.cc:301] APPLY: starting
0925 03:05:02.368111 (+ 97us) tablet.cc:1365] starting BulkCheckPresence
0925 03:05:02.368496 (+ 385us) tablet.cc:1368] finished BulkCheckPresence
0925 03:05:02.368517 (+ 21us) tablet.cc:1370] starting ApplyRowOperation cycle
0925 03:05:02.369283 (+ 766us) tablet.cc:1381] finished ApplyRowOperation cycle
0925 03:05:02.369532 (+ 249us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=4
0925 03:05:02.369584 (+ 52us) write_op.cc:312] APPLY: finished
0925 03:05:02.369875 (+ 291us) log.cc:842] Serialized 40 byte log entry
0925 03:05:02.447573 (+ 77698us) write_op.cc:489] Releasing partition, row and schema locks
0925 03:05:02.447691 (+ 118us) write_op.cc:454] Released schema lock
0925 03:05:02.447737 (+ 46us) write_op.cc:341] FINISH: Updating metrics
0925 03:05:02.448010 (+ 273us) write_op.cc:489] Releasing partition, row and schema locks
0925 03:05:02.448037 (+ 27us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":910149,"reported_tablets":10,"child_traces":[["op",{"apply.queue_time_us":2530760,"apply.run_cpu_time_us":11522,"apply.run_wall_time_us":80315,"num_ops":4,"prepare.queue_time_us":728180,"prepare.run_cpu_time_us":22460,"prepare.run_wall_time_us":503472,"raft.queue_time_us":343416,"raft.run_cpu_time_us":34162,"raft.run_wall_time_us":3294923,"replication_time_us":1946760,"spinlock_wait_cycles":151033984,"thread_start_us":4029527,"threads_started":5,"wal-append.queue_time_us":700409,"wal-append.run_cpu_time_us":44866,"wal-append.run_wall_time_us":3255927}]]}
W20260925 03:05:04.813340 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 247229us
W20260925 03:05:04.813000 25163 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 366330us
I20260925 03:05:04.825150 24369 catalog_manager.cc:5495] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "eeab93abfdb141a895002ec9f312296c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: HEALTHY } } }
I20260925 03:05:04.845381 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4ebd2b043024a548005c8b6b00b0449" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:04.859993 24539 raft_consensus.cc:1261] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260925 03:05:04.860221 24473 raft_consensus.cc:2379] T b4ebd2b043024a548005c8b6b00b0449 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 4: Already voted for candidate 1bc1f2e62ab44af995165bd5116a9d43 in this term.
I20260925 03:05:04.849679 24616 raft_consensus.cc:2454] T a4abe0ea757e4886be4945c5001e529a P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 3.
I20260925 03:05:04.811275 27094 leader_election.cc:288] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:04.854614 24540 raft_consensus.cc:3037] T 550118bc1a0641ce9803bdbf43f2069f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:05:04.861735 26800 raft_consensus.cc:729] T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:05:04.862661 26800 consensus_queue.cc:260] T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:04.863870 26800 raft_consensus.cc:3037] T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:05:04.857635 26946 raft_consensus.cc:2726] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260925 03:05:04.818564 27107 leader_election.cc:288] T edcf6e6a422d4d198d4fbb0f7d3d845d P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:04.870864 24540 raft_consensus.cc:2454] T 550118bc1a0641ce9803bdbf43f2069f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 7.
W20260925 03:05:04.874432 24616 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38564 (request call id 1386) took 1280 ms (client timeout 1532 ms). Trace:
I20260925 03:05:04.855151 24548 raft_consensus.cc:2454] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 6.
W20260925 03:05:04.868772 24436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.4:43979, user_credentials={real_user=slave}} blocked reactor thread for 54410.6us
I20260925 03:05:04.868878 27126 raft_consensus.cc:2781] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260925 03:05:04.858690 27077 raft_consensus.cc:2781] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260925 03:05:04.857700 27104 raft_consensus.cc:2726] T 3d89396576884f7c8901dba54f783947 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105) with higher term. Message: Invalid argument: T 3d89396576884f7c8901dba54f783947 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 4. Current term is 5.
I20260925 03:05:04.876152 25163 leader_election.cc:302] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5
W20260925 03:05:04.874837 24616 rpcz_store.cc:269] 0925 03:05:03.593330 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:05:04.774487 (+1181157us) service_pool.cc:227] Handling call
0925 03:05:04.873860 (+ 99373us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:05:04.880050 24511 leader_election.cc:302] T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 1bc1f2e62ab44af995165bd5116a9d43, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:04.877918 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ac285175464fbda67211d9574db415" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:05:04.881250 24539 raft_consensus.cc:3037] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:05:04.881918 25196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "293ef69caf514ac98510a68495f54eda" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:05:04.882565 25196 raft_consensus.cc:2454] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 5.
I20260925 03:05:04.883375 24511 leader_election.cc:398] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 election: Vote denied by peer 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979) with higher term. Message: Invalid argument: T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 5. Current term is 6.
I20260925 03:05:04.883675 24511 leader_election.cc:401] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20260925 03:05:04.885193 25198 raft_consensus.cc:1226] T e6b6ec010caf4689b8c90449764f5cf9 P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 5. Current term is 6. Ops: []
I20260925 03:05:04.889618 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f68804eb5a244d87bb395d50683c6012" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:04.891345 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e8af2641e124f9589c5ccfdac785f63" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:04.893070 24464 raft_consensus.cc:2354] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 6. Current term is 7.
I20260925 03:05:04.894774 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c23337a7cb74184a0e57b7095cb6a11" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:04.894831 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e54ca1757ee4e42956a9cebbeca4f62" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:04.896023 24471 raft_consensus.cc:2454] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 9.
I20260925 03:05:04.895740 24621 raft_consensus.cc:2454] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 6.
I20260925 03:05:04.891680 24511 leader_election.cc:302] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 1bc1f2e62ab44af995165bd5116a9d43, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:04.898196 24588 leader_election.cc:302] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:05:04.898692 24616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71f461a6e2ff434993b78a2d077f0abd" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:05:04.879174 27077 raft_consensus.cc:482] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:04.900147 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5332a5d329493991dff994e70ea92c" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:04.877272 27126 raft_consensus.cc:482] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:05:04.904256 24511 leader_election.cc:302] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 1bc1f2e62ab44af995165bd5116a9d43, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:04.886626 26811 raft_consensus.cc:2726] T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260925 03:05:04.911870 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "550118bc1a0641ce9803bdbf43f2069f" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:04.894963 26995 raft_consensus.cc:2781] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 FOLLOWER]: Leader election won for term 6
I20260925 03:05:04.915378 26995 raft_consensus.cc:686] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 LEADER]: Becoming Leader. State: Replica: 1bc1f2e62ab44af995165bd5116a9d43, State: Running, Role: LEADER
I20260925 03:05:04.916356 26995 consensus_queue.cc:237] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:04.908640 27126 raft_consensus.cc:3037] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Advancing to term 9
I20260925 03:05:04.892292 26937 raft_consensus.cc:2726] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979) with higher term. Message: Invalid argument: T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 5. Current term is 6.
I20260925 03:05:04.919988 24548 raft_consensus.cc:1261] T 442d4a29b1d743599f4b6b4871e26668 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260925 03:05:04.873979 27051 raft_consensus.cc:504] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:04.921342 25197 raft_consensus.cc:1226] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43 [term 10 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 9. Current term is 10. Ops: []
I20260925 03:05:04.919652 24613 raft_consensus.cc:1261] T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c [term 10 FOLLOWER]: Refusing update from remote peer 1bc1f2e62ab44af995165bd5116a9d43: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260925 03:05:04.924551 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e54ca1757ee4e42956a9cebbeca4f62" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:04.925276 24464 raft_consensus.cc:3037] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Advancing to term 10
I20260925 03:05:04.923079 25197 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49794 (request call id 272) took 1352 ms. Trace:
I20260925 03:05:04.926503 25197 rpcz_store.cc:276] 0925 03:05:03.570447 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:05:03.571352 (+ 905us) service_pool.cc:227] Handling call
0925 03:05:04.667362 (+1096010us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:05:04.921731 (+254369us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:05:04.922364 (+ 633us) spinlock_profiling.cc:243] Waited 254 ms on lock 0x7b5400433ad4. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d0f51 00007f43764cf99e 00007f437b4112e9 00007f4374af7684 00007f4374af7612 00007f4374af7541 00007f4374af74bc 00007f4374af6772 00007f43748fa94c 00007f43748f9d86
0925 03:05:04.922873 (+ 509us) spinlock_profiling.cc:243] Waited 1.09 s on lock 0x7b5400433ad0. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764cfa09 00007f437b4112e9 00007f4374af7684 00007f4374af7612 00007f4374af7541 00007f4374af74bc 00007f4374af6772 00007f43748fa94c 00007f43748f9d86 00007f43748f972f
0925 03:05:04.923020 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3097250432}
I20260925 03:05:04.928229 26900 consensus_queue.cc:1035] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260925 03:05:04.938797 24511 leader_election.cc:302] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:05:04.941372 24539 raft_consensus.cc:2454] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 7.
I20260925 03:05:04.943022 24539 raft_consensus.cc:1261] T 3d89396576884f7c8901dba54f783947 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260925 03:05:04.936082 24586 leader_election.cc:302] T 550118bc1a0641ce9803bdbf43f2069f P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:05:04.909250 27077 raft_consensus.cc:3037] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Advancing to term 10
I20260925 03:05:04.944777 24369 catalog_manager.cc:5495] T 100c113a9fd84cc8adf2ec7b33ea0093 P eeab93abfdb141a895002ec9f312296c reported cstate change: term changed from 1 to 3, leader changed from 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1) to eeab93abfdb141a895002ec9f312296c (127.22.193.3). New cstate: current_term: 3 leader_uuid: "eeab93abfdb141a895002ec9f312296c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: HEALTHY } } }
I20260925 03:05:04.953878 24465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b627ecb4a5384ef392aa695e06599c39" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:04.956980 24465 raft_consensus.cc:2454] T b627ecb4a5384ef392aa695e06599c39 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 6.
I20260925 03:05:04.971474 24464 raft_consensus.cc:2454] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 10.
I20260925 03:05:04.972419 24471 raft_consensus.cc:1261] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [term 10 FOLLOWER]: Refusing update from remote peer 1bc1f2e62ab44af995165bd5116a9d43: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260925 03:05:04.957394 27071 raft_consensus.cc:2781] T 550118bc1a0641ce9803bdbf43f2069f P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Leader election won for term 7
W20260925 03:05:04.976166 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 89227.3us
I20260925 03:05:04.978134 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9c33fd100ff44c1a3247319a324a044" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:04.955056 26937 raft_consensus.cc:2781] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260925 03:05:04.978956 24464 raft_consensus.cc:2354] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 6. Current term is 7.
I20260925 03:05:04.954039 27190 raft_consensus.cc:2726] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260925 03:05:04.971339 25197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:05:04.966749 27041 raft_consensus.cc:2726] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260925 03:05:04.993851 24616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e725b82985746b39c0cc8f671c95353" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:04.989236 27071 raft_consensus.cc:686] T 550118bc1a0641ce9803bdbf43f2069f P eeab93abfdb141a895002ec9f312296c [term 7 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:05:05.005915 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43366a0a2f9d4344bf4e37ffd290ef8a" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:05.006796 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcf6e6a422d4d198d4fbb0f7d3d845d" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:05.009640 24436 leader_election.cc:302] T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: 1bc1f2e62ab44af995165bd5116a9d43, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:05.014153 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ac285175464fbda67211d9574db415" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:05:05.012871 24586 leader_election.cc:302] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
W20260925 03:05:05.017371 24541 consensus_meta.cc:287] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent flushing consensus metadata: real 0.535s user 0.007s sys 0.009s
I20260925 03:05:05.021356 24615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:05.021848 24436 leader_election.cc:302] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: 1bc1f2e62ab44af995165bd5116a9d43, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:04.981917 26937 raft_consensus.cc:482] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:05:04.970331 26946 consensus_queue.cc:1035] T c88cbd39e01f44628a094af516be2833 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.032s
I20260925 03:05:05.025668 26979 consensus_queue.cc:1046] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.071s
I20260925 03:05:05.026552 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bb46bce3564841b7674e6a693bd2e6" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:05.046275 24464 raft_consensus.cc:2454] T 11bb46bce3564841b7674e6a693bd2e6 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 3.
I20260925 03:05:05.030130 27071 consensus_queue.cc:237] T 550118bc1a0641ce9803bdbf43f2069f P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:05:05.031530 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 48993us
I20260925 03:05:05.036240 27051 leader_election.cc:288] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 10 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:05.042542 24369 catalog_manager.cc:5495] T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c reported cstate change: term changed from 3 to 5, leader changed from a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2) to eeab93abfdb141a895002ec9f312296c (127.22.193.3). New cstate: current_term: 5 leader_uuid: "eeab93abfdb141a895002ec9f312296c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: HEALTHY } } }
I20260925 03:05:05.043787 24613 raft_consensus.cc:2379] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 in current term 7: Already voted for candidate eeab93abfdb141a895002ec9f312296c in this term.
I20260925 03:05:05.017762 24541 raft_consensus.cc:2454] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 8.
I20260925 03:05:05.054087 24548 raft_consensus.cc:1226] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 7. Current term is 8. Ops: []
I20260925 03:05:05.057601 24436 leader_election.cc:302] T 9e8af2641e124f9589c5ccfdac785f63 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:05:05.068292 24615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5aec38e66a947c681e6c2a72b05e6ca" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:05:05.069227 25163 leader_election.cc:302] T 71f461a6e2ff434993b78a2d077f0abd P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:05:05.014246 27094 raft_consensus.cc:2726] T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260925 03:05:05.041173 26937 raft_consensus.cc:3037] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:05:04.990832 27107 consensus_queue.cc:1046] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.062s
I20260925 03:05:05.011659 27167 consensus_queue.cc:1035] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.059s
I20260925 03:05:05.002194 27009 consensus_queue.cc:1046] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.068s
I20260925 03:05:05.088833 27022 raft_consensus.cc:3032] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [term 9 LEADER]: Stepping down as leader of term 9
I20260925 03:05:05.089282 27022 raft_consensus.cc:729] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [term 9 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:05:05.089409 27107 raft_consensus.cc:3032] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 LEADER]: Stepping down as leader of term 7
I20260925 03:05:05.089921 27107 raft_consensus.cc:729] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:05:05.090106 27022 consensus_queue.cc:260] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:05.091399 27022 raft_consensus.cc:3037] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Advancing to term 10
I20260925 03:05:05.091769 27112 consensus_queue.cc:1035] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260925 03:05:05.090829 27107 consensus_queue.cc:260] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:05.092919 27107 raft_consensus.cc:3037] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Advancing to term 8
I20260925 03:05:05.094605 24465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5332a5d329493991dff994e70ea92c" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:05.095232 24465 raft_consensus.cc:2454] T 4e5332a5d329493991dff994e70ea92c P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 7.
I20260925 03:05:05.096098 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367277daf8d04778b3fa4ed94d71580d" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:05.097078 24471 raft_consensus.cc:1226] T e6b6ec010caf4689b8c90449764f5cf9 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 5. Current term is 6. Ops: [5.8-5.8]
I20260925 03:05:05.097254 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15861cd28e354598a3c6bfc4337c34bb" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:05.097739 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60d45b7d467d4e1ab92367f6d7d52b15" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:05.097988 24472 raft_consensus.cc:2354] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 5. Current term is 6.
I20260925 03:05:05.098351 24473 raft_consensus.cc:2454] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 8.
I20260925 03:05:05.098940 24465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b5df303cd6a4d59aca5c18bbbff24ed" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:05.099620 24467 raft_consensus.cc:1226] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 6. Current term is 7. Ops: [4.4-6.5]
I20260925 03:05:05.099570 24465 raft_consensus.cc:2454] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 8.
I20260925 03:05:05.055140 27064 consensus_queue.cc:1035] T 3d89396576884f7c8901dba54f783947 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.046s
I20260925 03:05:05.100730 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35992316cb8d4c028333388eff02ecde" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:05.101572 24468 raft_consensus.cc:2379] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 7: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:05:05.102286 26741 raft_consensus.cc:2781] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Leader election won for term 7
I20260925 03:05:05.102865 26741 raft_consensus.cc:686] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [term 7 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:05:05.104072 24622 raft_consensus.cc:2454] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 8.
I20260925 03:05:05.103793 26741 consensus_queue.cc:237] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:05.104856 27185 raft_consensus.cc:2781] T 9e8af2641e124f9589c5ccfdac785f63 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260925 03:05:05.105324 27185 raft_consensus.cc:482] T 9e8af2641e124f9589c5ccfdac785f63 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting leader election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:05:05.105657 27185 raft_consensus.cc:3037] T 9e8af2641e124f9589c5ccfdac785f63 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:05:05.089744 26803 raft_consensus.cc:2781] T 71f461a6e2ff434993b78a2d077f0abd P 1bc1f2e62ab44af995165bd5116a9d43 [term 8 FOLLOWER]: Leader election won for term 8
I20260925 03:05:05.106345 26803 raft_consensus.cc:686] T 71f461a6e2ff434993b78a2d077f0abd P 1bc1f2e62ab44af995165bd5116a9d43 [term 8 LEADER]: Becoming Leader. State: Replica: 1bc1f2e62ab44af995165bd5116a9d43, State: Running, Role: LEADER
I20260925 03:05:05.107249 26803 consensus_queue.cc:237] T 71f461a6e2ff434993b78a2d077f0abd P 1bc1f2e62ab44af995165bd5116a9d43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:05.060782 27116 raft_consensus.cc:2726] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260925 03:05:05.121546 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5aec38e66a947c681e6c2a72b05e6ca" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:05.122720 24622 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38564 (request call id 1412) took 1032 ms. Trace:
I20260925 03:05:05.123139 24622 rpcz_store.cc:276] 0925 03:05:04.090481 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:05:04.102832 (+ 12351us) service_pool.cc:227] Handling call
0925 03:05:04.741652 (+638820us) spinlock_profiling.cc:243] Waited 7.55 ms on lock 0x7b58001d3d98. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f43733ecf11 00007f4373570851 00007f4373570702 00007f437c460b09 00007f437b39c730 00007f437b46279b 00007f437b462821 00007f437b4121b0 00007f437b411a94 00007f4374af9c04 00007f4374af9b92
0925 03:05:05.122528 (+380876us) spinlock_profiling.cc:243] Waited 357 ms on lock 0x7b5400217fd4. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d42ae 00007f437b411c27 00007f4374af9c04 00007f4374af9b92 00007f4374af9ac1 00007f4374af9a3c 00007f4374af8cf2 00007f43748fa94c 00007f43748f9d86 00007f43748f972f
0925 03:05:05.122652 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":838795776}
I20260925 03:05:05.016163 27053 consensus_queue.cc:1035] T 442d4a29b1d743599f4b6b4871e26668 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.062s
I20260925 03:05:05.069176 25197 raft_consensus.cc:2396] T 29637f5a4a504ce6ba29033ccd0cd716 P 1bc1f2e62ab44af995165bd5116a9d43 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 for term 9 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20260925 03:05:05.055811 27104 raft_consensus.cc:2726] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260925 03:05:05.137382 26811 raft_consensus.cc:3032] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 LEADER]: Stepping down as leader of term 5
I20260925 03:05:05.137856 26811 raft_consensus.cc:729] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:05:05.138742 26811 consensus_queue.cc:260] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 5.8, Last appended by leader: 8, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:05.140154 26811 raft_consensus.cc:3037] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:05:05.109742 24369 catalog_manager.cc:5495] T c88cbd39e01f44628a094af516be2833 P eeab93abfdb141a895002ec9f312296c reported cstate change: term changed from 3 to 5, leader changed from a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2) to eeab93abfdb141a895002ec9f312296c (127.22.193.3). New cstate: current_term: 5 leader_uuid: "eeab93abfdb141a895002ec9f312296c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: HEALTHY } } }
I20260925 03:05:05.150030 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b8e0d260a884ecdb20f79de46c36b19" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
W20260925 03:05:05.157991 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 74743.1us
I20260925 03:05:05.159202 25165 leader_election.cc:302] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:05.163080 24369 catalog_manager.cc:5495] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c reported cstate change: term changed from 1 to 5. New cstate: current_term: 5 leader_uuid: "eeab93abfdb141a895002ec9f312296c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: HEALTHY } } }
I20260925 03:05:05.164572 24436 leader_election.cc:302] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: 1bc1f2e62ab44af995165bd5116a9d43, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:05.177332 24369 catalog_manager.cc:5495] T 37a157996e944db2906110d87ecb0c54 P eeab93abfdb141a895002ec9f312296c reported cstate change: term changed from 4 to 5, leader changed from 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1) to eeab93abfdb141a895002ec9f312296c (127.22.193.3). New cstate: current_term: 5 leader_uuid: "eeab93abfdb141a895002ec9f312296c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: HEALTHY } } }
I20260925 03:05:05.160797 27167 raft_consensus.cc:2726] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260925 03:05:05.199821 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e5819c15eac4544a57e5ee6c9d4f117" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:05.201457 24469 raft_consensus.cc:2454] T 8e5819c15eac4544a57e5ee6c9d4f117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 7.
I20260925 03:05:05.194011 24369 catalog_manager.cc:5495] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c reported cstate change: term changed from 2 to 5. New cstate: current_term: 5 leader_uuid: "eeab93abfdb141a895002ec9f312296c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: HEALTHY } } }
I20260925 03:05:05.205986 24588 leader_election.cc:302] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters: 1bc1f2e62ab44af995165bd5116a9d43
I20260925 03:05:05.215543 26909 consensus_queue.cc:1046] T 71f461a6e2ff434993b78a2d077f0abd P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 5, Time since last communication: 0.016s
I20260925 03:05:05.188418 26946 raft_consensus.cc:2726] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260925 03:05:05.236342 24467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21c43af5e6844fdaacc8c625fa01fd8d" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:05.237046 24467 raft_consensus.cc:3037] T 21c43af5e6844fdaacc8c625fa01fd8d P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:05:05.239310 24465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be4e0aa984f4f2b81a58e382e434ad5" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:05.240111 24465 raft_consensus.cc:2379] T 0be4e0aa984f4f2b81a58e382e434ad5 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 6: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:05:05.240919 24465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:05.241766 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7993f9c3e2a44f408bbf5d1e395e2306" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:05.242868 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f1fac028a144058b92331c0d9b44ebf" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:05.243400 24472 raft_consensus.cc:2454] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 7.
I20260925 03:05:05.244096 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c23337a7cb74184a0e57b7095cb6a11" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:05.245086 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b030d84c670b48d5bff337ccc0c77626" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:05.246402 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c85f5d1d3f004cf5817bb594fd8e6469" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:05.244633 24472 raft_consensus.cc:2454] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 7.
I20260925 03:05:05.255587 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48ca6ccd98194cf2af7b5278d15bf4ea" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:05.255183 24465 raft_consensus.cc:2454] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 8.
I20260925 03:05:05.257958 24467 raft_consensus.cc:2454] T 21c43af5e6844fdaacc8c625fa01fd8d P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 5.
I20260925 03:05:05.262084 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37a157996e944db2906110d87ecb0c54" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
W20260925 03:05:05.262539 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 76527.8us
I20260925 03:05:05.267808 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74443a0a1dc34d88a2f42e3b722db405" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:05.271958 24469 raft_consensus.cc:2454] T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 5.
I20260925 03:05:05.261847 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "100c113a9fd84cc8adf2ec7b33ea0093" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:05.261942 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:05.274915 24470 raft_consensus.cc:2379] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 8: Already voted for candidate eeab93abfdb141a895002ec9f312296c in this term.
I20260925 03:05:05.262813 24472 raft_consensus.cc:2379] T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 5: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:05:05.258994 24369 catalog_manager.cc:5495] T b1e85248d3b04b979dd080caeeb02fdb P eeab93abfdb141a895002ec9f312296c reported cstate change: term changed from 1 to 3, leader changed from 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1) to eeab93abfdb141a895002ec9f312296c (127.22.193.3). New cstate: current_term: 3 leader_uuid: "eeab93abfdb141a895002ec9f312296c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260925 03:05:05.261576 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43366a0a2f9d4344bf4e37ffd290ef8a" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:05.269601 24473 raft_consensus.cc:2454] T 37a157996e944db2906110d87ecb0c54 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 7.
I20260925 03:05:05.285216 24467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e725b82985746b39c0cc8f671c95353" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:05.284881 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7aa47c48404465b94a36fd819fe3c7a" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:05.290124 24467 raft_consensus.cc:2454] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 6.
I20260925 03:05:05.290220 24472 raft_consensus.cc:2454] T a7aa47c48404465b94a36fd819fe3c7a P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 6.
I20260925 03:05:05.258157 27185 raft_consensus.cc:504] T 9e8af2641e124f9589c5ccfdac785f63 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:05.300320 24468 raft_consensus.cc:2379] T 43366a0a2f9d4344bf4e37ffd290ef8a P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 4: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:05:05.261284 27146 raft_consensus.cc:2781] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260925 03:05:05.313895 24613 raft_consensus.cc:1226] T 550118bc1a0641ce9803bdbf43f2069f P eeab93abfdb141a895002ec9f312296c [term 7 LEADER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 6. Current term is 7. Ops: []
I20260925 03:05:05.337963 24466 raft_consensus.cc:1226] T 9e8af2641e124f9589c5ccfdac785f63 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 6. Current term is 7. Ops: [6.6-6.6]
I20260925 03:05:05.345093 24369 catalog_manager.cc:5495] T 7993f9c3e2a44f408bbf5d1e395e2306 P eeab93abfdb141a895002ec9f312296c reported cstate change: term changed from 1 to 3, leader changed from a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2) to eeab93abfdb141a895002ec9f312296c (127.22.193.3). New cstate: current_term: 3 leader_uuid: "eeab93abfdb141a895002ec9f312296c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: HEALTHY } } }
I20260925 03:05:05.349423 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5278a60f97da4e10aecd50a8d9a9a43c" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:05.351894 24468 raft_consensus.cc:2379] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 5: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:05:05.322566 27146 raft_consensus.cc:482] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Starting leader election (detected failure of leader 1bc1f2e62ab44af995165bd5116a9d43)
I20260925 03:05:05.360414 24588 leader_election.cc:302] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:05:05.363040 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25102ba08e4e49388f6176badcd10a25" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:05.363873 24469 raft_consensus.cc:2454] T 25102ba08e4e49388f6176badcd10a25 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 5.
I20260925 03:05:05.365159 26756 raft_consensus.cc:3032] T 71f461a6e2ff434993b78a2d077f0abd P eeab93abfdb141a895002ec9f312296c [term 7 LEADER]: Stepping down as leader of term 7
I20260925 03:05:05.365594 26756 raft_consensus.cc:729] T 71f461a6e2ff434993b78a2d077f0abd P eeab93abfdb141a895002ec9f312296c [term 7 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:05:05.366421 26756 consensus_queue.cc:260] T 71f461a6e2ff434993b78a2d077f0abd P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:05.367820 26756 raft_consensus.cc:3037] T 71f461a6e2ff434993b78a2d077f0abd P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Advancing to term 8
I20260925 03:05:05.373257 26577 raft_consensus.cc:2781] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260925 03:05:05.373713 26577 raft_consensus.cc:482] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:05:05.374054 26577 raft_consensus.cc:3037] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:05:05.374194 24588 leader_election.cc:302] T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:05:05.375072 26842 raft_consensus.cc:2726] T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260925 03:05:05.376122 27077 raft_consensus.cc:504] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:05.376554 27064 consensus_queue.cc:1046] T 43366a0a2f9d4344bf4e37ffd290ef8a P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260925 03:05:05.378139 27041 raft_consensus.cc:3032] T 43366a0a2f9d4344bf4e37ffd290ef8a P eeab93abfdb141a895002ec9f312296c [term 3 LEADER]: Stepping down as leader of term 3
I20260925 03:05:05.378598 27041 raft_consensus.cc:729] T 43366a0a2f9d4344bf4e37ffd290ef8a P eeab93abfdb141a895002ec9f312296c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:05:05.379120 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcf6e6a422d4d198d4fbb0f7d3d845d" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:05.379846 25198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5aec38e66a947c681e6c2a72b05e6ca" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43"
I20260925 03:05:05.380564 25198 raft_consensus.cc:2379] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 in current term 10: Already voted for candidate 1bc1f2e62ab44af995165bd5116a9d43 in this term.
I20260925 03:05:05.380604 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9357d144aa49789faaf31fe5e6bf89" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:05.381022 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5f9b503b0ab43999d582015a367441e" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:05.381774 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b627ecb4a5384ef392aa695e06599c39" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:05.382131 24465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c88cbd39e01f44628a094af516be2833" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:05.382332 24472 raft_consensus.cc:2454] T b627ecb4a5384ef392aa695e06599c39 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 6.
I20260925 03:05:05.383214 24615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5aec38e66a947c681e6c2a72b05e6ca" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:05:05.383738 24467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bb46bce3564841b7674e6a693bd2e6" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:05.384334 24467 raft_consensus.cc:2454] T 11bb46bce3564841b7674e6a693bd2e6 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 3.
I20260925 03:05:05.385041 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "420163a6cdce412ebc21c25fe3835638" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:05.385582 27185 leader_election.cc:288] T 9e8af2641e124f9589c5ccfdac785f63 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:05.386176 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "550118bc1a0641ce9803bdbf43f2069f" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:05.379413 27041 consensus_queue.cc:260] T 43366a0a2f9d4344bf4e37ffd290ef8a P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:05.391059 24615 raft_consensus.cc:2454] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 10.
I20260925 03:05:05.391168 26909 consensus_queue.cc:1046] T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260925 03:05:05.391465 27022 consensus_queue.cc:1035] T 0b5df303cd6a4d59aca5c18bbbff24ed P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20260925 03:05:05.392267 26909 raft_consensus.cc:3032] T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c [term 5 LEADER]: Stepping down as leader of term 5
I20260925 03:05:05.392624 26909 raft_consensus.cc:729] T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c [term 5 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:05:05.393356 24588 leader_election.cc:302] T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:05:05.383282 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1414dcfe53f64a78ae413531eb8ec117" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:05.394892 24470 raft_consensus.cc:2396] T 1414dcfe53f64a78ae413531eb8ec117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260925 03:05:05.389885 27146 raft_consensus.cc:3037] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:05:05.393689 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceab6b0fccd54e909a64f66fe81a9a1f" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:05.401664 24469 raft_consensus.cc:2379] T ceab6b0fccd54e909a64f66fe81a9a1f P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 7: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
W20260925 03:05:05.399674 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 134620us
I20260925 03:05:05.394222 27071 raft_consensus.cc:2781] T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Leader election won for term 8
I20260925 03:05:05.412982 24436 leader_election.cc:302] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters: 1bc1f2e62ab44af995165bd5116a9d43
I20260925 03:05:05.422546 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abd5b8dc1b7e49e3aa9af900d73ff0e7" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:05.415844 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e8af2641e124f9589c5ccfdac785f63" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:05:05.423357 24466 raft_consensus.cc:2454] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 6.
I20260925 03:05:05.393435 26909 consensus_queue.cc:260] T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:05.426702 27104 consensus_queue.cc:1046] T 550118bc1a0641ce9803bdbf43f2069f P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260925 03:05:05.427565 27104 consensus_queue.cc:1225] T 550118bc1a0641ce9803bdbf43f2069f P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Peer eeab93abfdb141a895002ec9f312296c log is divergent from this leader: its last log entry 7.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260925 03:05:05.427645 26909 raft_consensus.cc:3037] T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:05:05.428102 27104 raft_consensus.cc:3032] T 550118bc1a0641ce9803bdbf43f2069f P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 LEADER]: Stepping down as leader of term 6
I20260925 03:05:05.428448 27104 raft_consensus.cc:729] T 550118bc1a0641ce9803bdbf43f2069f P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:05:05.429227 27104 consensus_queue.cc:260] T 550118bc1a0641ce9803bdbf43f2069f P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:05.430394 27104 raft_consensus.cc:3037] T 550118bc1a0641ce9803bdbf43f2069f P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:05:05.433363 24467 raft_consensus.cc:1261] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Refusing update from remote peer 1bc1f2e62ab44af995165bd5116a9d43: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260925 03:05:05.434747 26937 raft_consensus.cc:504] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:05.435395 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442d4a29b1d743599f4b6b4871e26668" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:05.436084 24470 raft_consensus.cc:2454] T 442d4a29b1d743599f4b6b4871e26668 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 7.
I20260925 03:05:05.378320 27077 leader_election.cc:288] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 10 election: Requested vote from peers eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:05:05.478832 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9c33fd100ff44c1a3247319a324a044" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
W20260925 03:05:05.488487 25165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 47496.7us
I20260925 03:05:05.440258 27185 raft_consensus.cc:2781] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [term 10 FOLLOWER]: Leader election won for term 10
I20260925 03:05:05.486140 24544 raft_consensus.cc:1261] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Refusing update from remote peer 1bc1f2e62ab44af995165bd5116a9d43: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260925 03:05:05.495921 24469 raft_consensus.cc:2354] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 6. Current term is 7.
W20260925 03:05:05.497625 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 58265.2us
I20260925 03:05:05.506045 24369 catalog_manager.cc:5495] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c reported cstate change: term changed from 3 to 5, leader changed from 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1) to eeab93abfdb141a895002ec9f312296c (127.22.193.3). New cstate: current_term: 5 leader_uuid: "eeab93abfdb141a895002ec9f312296c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: HEALTHY } } }
I20260925 03:05:05.511590 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60d45b7d467d4e1ab92367f6d7d52b15" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:05.515985 24471 raft_consensus.cc:2454] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 8.
I20260925 03:05:05.517619 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc2df40db08649c9a75e90000b878559" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:05.501580 27185 raft_consensus.cc:686] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [term 10 LEADER]: Becoming Leader. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:05:05.519547 27185 consensus_queue.cc:237] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:05.520973 24588 leader_election.cc:302] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:05:05.524109 27041 raft_consensus.cc:3037] T 43366a0a2f9d4344bf4e37ffd290ef8a P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:05:05.504419 27071 raft_consensus.cc:686] T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c [term 8 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:05:05.531060 26842 raft_consensus.cc:2781] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Leader election won for term 7
I20260925 03:05:05.531028 27071 consensus_queue.cc:237] T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 8, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:05.531915 26842 raft_consensus.cc:686] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c [term 7 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:05:05.533072 26842 consensus_queue.cc:237] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:05.539522 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367277daf8d04778b3fa4ed94d71580d" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:05.544338 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35992316cb8d4c028333388eff02ecde" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:05.545315 24470 raft_consensus.cc:2379] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 7: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:05:05.544582 26811 raft_consensus.cc:482] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:05:05.545924 26811 raft_consensus.cc:504] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:05.495428 26803 consensus_queue.cc:1035] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.047s
I20260925 03:05:05.548635 24545 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33068 (request call id 180) took 1088 ms. Trace:
I20260925 03:05:05.549068 24545 rpcz_store.cc:276] 0925 03:05:04.459821 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:05:04.459963 (+ 142us) service_pool.cc:227] Handling call
0925 03:05:04.472484 (+ 12521us) raft_consensus.cc:1457] Updating replica for 2 ops
0925 03:05:04.820024 (+347540us) spinlock_profiling.cc:243] Waited 305 ms on lock 0x7b180037d380. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f437c4478b1 00007f437c454661 00007f43735a8638 00007f43735abb11 00007f43735abac9 00007f43735aba59 00007f43735aba21 00007f43735aad1d 00007f437c80c204 00007f437c80c039 00007f437358adc6
0925 03:05:05.166028 (+346004us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:05:05.166063 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0925 03:05:05.254276 (+ 88213us) spinlock_profiling.cc:243] Waited 34.9 ms on lock 0x7b2c00000238. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4374eb3e11 00007f4374ec3442 00007f437650bc11 00007f43764d058a 00007f43764cf99e 00007f437b4112e9 00007f4374af7684 00007f4374af7612 00007f4374af7541 00007f4374af74bc 00007f4374af6772
0925 03:05:05.322249 (+ 67973us) log.cc:842] Serialized 52 byte log entry
0925 03:05:05.440062 (+117813us) raft_consensus.cc:1631] Marking committed up to 8
0925 03:05:05.479769 (+ 39707us) log.cc:842] Serialized 14 byte log entry
0925 03:05:05.480156 (+ 387us) log.cc:842] Serialized 14 byte log entry
0925 03:05:05.480318 (+ 162us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:05:05.480388 (+ 70us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:05:05.548427 (+ 68039us) raft_consensus.cc:1671] finished
0925 03:05:05.548446 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:05:05.548579 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5250,"prepare.queue_time_us":39642,"prepare.run_cpu_time_us":61,"prepare.run_wall_time_us":56,"raft.queue_time_us":26458,"raft.run_cpu_time_us":202,"raft.run_wall_time_us":209,"spinlock_wait_cycles":782269568,"thread_start_us":431072,"threads_started":2,"wal-append.queue_time_us":206035}
I20260925 03:05:05.548166 26811 leader_election.cc:288] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:05.559580 24369 catalog_manager.cc:5495] T ac9357d144aa49789faaf31fe5e6bf89 P eeab93abfdb141a895002ec9f312296c reported cstate change: term changed from 1 to 2, leader changed from 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1) to eeab93abfdb141a895002ec9f312296c (127.22.193.3). New cstate: current_term: 2 leader_uuid: "eeab93abfdb141a895002ec9f312296c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: HEALTHY } } }
I20260925 03:05:05.564605 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06f4a7eaf6534e4986c9fd35911038e4" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:05.568958 24622 raft_consensus.cc:1261] T 71f461a6e2ff434993b78a2d077f0abd P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Refusing update from remote peer 1bc1f2e62ab44af995165bd5116a9d43: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260925 03:05:05.570616 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "293ef69caf514ac98510a68495f54eda" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:05.577306 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5332a5d329493991dff994e70ea92c" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:05.577894 24467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15861cd28e354598a3c6bfc4337c34bb" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:05.578229 24470 raft_consensus.cc:2454] T 4e5332a5d329493991dff994e70ea92c P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 7.
I20260925 03:05:05.578641 24467 raft_consensus.cc:2354] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 5. Current term is 6.
I20260925 03:05:05.581055 24369 catalog_manager.cc:5495] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "eeab93abfdb141a895002ec9f312296c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: HEALTHY } } }
I20260925 03:05:05.581569 26937 leader_election.cc:288] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 6 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:05:05.583451 25198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "293ef69caf514ac98510a68495f54eda" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43"
I20260925 03:05:05.584048 25198 raft_consensus.cc:3037] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43 [term 5 FOLLOWER]: Advancing to term 6
W20260925 03:05:05.581717 27126 consensus_meta.cc:287] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent flushing consensus metadata: real 0.664s user 0.012s sys 0.000s
I20260925 03:05:05.598953 24369 catalog_manager.cc:5495] T a7aa47c48404465b94a36fd819fe3c7a P eeab93abfdb141a895002ec9f312296c reported cstate change: term changed from 1 to 2, leader changed from 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1) to eeab93abfdb141a895002ec9f312296c (127.22.193.3). New cstate: current_term: 2 leader_uuid: "eeab93abfdb141a895002ec9f312296c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: HEALTHY } } }
I20260925 03:05:05.600797 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43366a0a2f9d4344bf4e37ffd290ef8a" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:05.601048 24467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b8e0d260a884ecdb20f79de46c36b19" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:05.601686 24466 raft_consensus.cc:2379] T 43366a0a2f9d4344bf4e37ffd290ef8a P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 4: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:05:05.602007 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21c43af5e6844fdaacc8c625fa01fd8d" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:05.602638 24472 raft_consensus.cc:2362] T 21c43af5e6844fdaacc8c625fa01fd8d P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 5. Re-sending same reply.
I20260925 03:05:05.602722 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37a157996e944db2906110d87ecb0c54" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:05.603359 24469 raft_consensus.cc:2454] T 37a157996e944db2906110d87ecb0c54 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 7.
I20260925 03:05:05.584344 27146 raft_consensus.cc:504] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:05.603592 24467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f1fac028a144058b92331c0d9b44ebf" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:05.604230 24467 raft_consensus.cc:2454] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 7.
I20260925 03:05:05.562752 24615 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "04ac285175464fbda67211d9574db415"
dest_uuid: "eeab93abfdb141a895002ec9f312296c"
mode: GRACEFUL
new_leader_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
from {username='slave'} at 127.0.0.1:44570
I20260925 03:05:05.605124 24620 raft_consensus.cc:1226] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Rejecting Update request from peer 1bc1f2e62ab44af995165bd5116a9d43 for earlier term 5. Current term is 6. Ops: []
I20260925 03:05:05.605250 24615 raft_consensus.cc:595] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [term 7 LEADER]: Received request to transfer leadership to TS a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:05:05.606765 27146 leader_election.cc:288] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 6 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:05:05.609076 25197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c81a1b3b616492ab6c964f45af36077" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43"
I20260925 03:05:05.606779 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e5819c15eac4544a57e5ee6c9d4f117" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:05.593120 26995 consensus_queue.cc:1035] T 71f461a6e2ff434993b78a2d077f0abd P 1bc1f2e62ab44af995165bd5116a9d43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.013s
I20260925 03:05:05.603942 24465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "100c113a9fd84cc8adf2ec7b33ea0093" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:05.611650 24466 raft_consensus.cc:2454] T 8e5819c15eac4544a57e5ee6c9d4f117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 7.
I20260925 03:05:05.603513 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48ca6ccd98194cf2af7b5278d15bf4ea" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:05.603029 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74443a0a1dc34d88a2f42e3b722db405" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:05.612550 24464 raft_consensus.cc:2454] T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 5.
I20260925 03:05:05.612571 24473 raft_consensus.cc:2379] T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 5: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
W20260925 03:05:05.590167 24514 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.22.193.1:42187 timed out after 2.244s (SENT) [suppressed 38 similar messages]
I20260925 03:05:05.617493 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b030d84c670b48d5bff337ccc0c77626" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:05.604184 27126 raft_consensus.cc:504] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:05.620649 24617 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8ac466793d094325ace224708b2158ed"
dest_uuid: "eeab93abfdb141a895002ec9f312296c"
mode: GRACEFUL
new_leader_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
from {username='slave'} at 127.0.0.1:44570
I20260925 03:05:05.621269 24617 raft_consensus.cc:595] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Received request to transfer leadership to TS a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:05:05.621603 24617 raft_consensus.cc:603] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260925 03:05:05.623683 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 61217.8us
I20260925 03:05:05.566424 27167 consensus_queue.cc:1035] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.030s
I20260925 03:05:05.631261 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:05.632203 24470 raft_consensus.cc:2454] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 8.
I20260925 03:05:05.639848 24545 raft_consensus.cc:1261] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Refusing update from remote peer 1bc1f2e62ab44af995165bd5116a9d43: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260925 03:05:05.640744 26762 consensus_queue.cc:1035] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260925 03:05:05.640877 27022 consensus_queue.cc:1035] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
W20260925 03:05:05.641928 27217 log.cc:924] Time spent T 3d89396576884f7c8901dba54f783947 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.066s user 0.001s sys 0.000s
I20260925 03:05:05.650228 26803 consensus_queue.cc:1046] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20260925 03:05:05.651361 27207 log.cc:924] Time spent T 0e54ca1757ee4e42956a9cebbeca4f62 P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.213s user 0.003s sys 0.000s
I20260925 03:05:05.652191 27126 leader_election.cc:288] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 9 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:05.656168 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7691c6b475428c82d6181d693d26f3" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:05.657439 25198 raft_consensus.cc:2454] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 6.
I20260925 03:05:05.658754 24511 leader_election.cc:302] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:05:05.657020 24470 raft_consensus.cc:2454] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 8.
W20260925 03:05:05.661985 27214 log.cc:924] Time spent T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.227s user 0.003s sys 0.000s
I20260925 03:05:05.663938 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be4e0aa984f4f2b81a58e382e434ad5" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:05.651331 26803 raft_consensus.cc:3032] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43 [term 5 LEADER]: Stepping down as leader of term 5
I20260925 03:05:05.664822 24470 raft_consensus.cc:2379] T 0be4e0aa984f4f2b81a58e382e434ad5 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 6: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:05:05.665719 24465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "293ef69caf514ac98510a68495f54eda" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:05.666956 24620 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a4abe0ea757e4886be4945c5001e529a"
dest_uuid: "eeab93abfdb141a895002ec9f312296c"
mode: GRACEFUL
new_leader_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
from {username='slave'} at 127.0.0.1:44570
I20260925 03:05:05.668300 24620 raft_consensus.cc:595] T a4abe0ea757e4886be4945c5001e529a P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Received request to transfer leadership to TS a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:05:05.668649 24620 raft_consensus.cc:603] T a4abe0ea757e4886be4945c5001e529a P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260925 03:05:05.698642 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7993f9c3e2a44f408bbf5d1e395e2306" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:05.700982 24465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5f9b503b0ab43999d582015a367441e" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:05.666038 26803 raft_consensus.cc:729] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1bc1f2e62ab44af995165bd5116a9d43, State: Running, Role: LEADER
I20260925 03:05:05.667091 27126 raft_consensus.cc:2781] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader election won for term 6
I20260925 03:05:05.660957 27167 consensus_queue.cc:1035] T 71f461a6e2ff434993b78a2d077f0abd P 1bc1f2e62ab44af995165bd5116a9d43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.036s
I20260925 03:05:05.710558 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f68804eb5a244d87bb395d50683c6012" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:05.711352 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60d45b7d467d4e1ab92367f6d7d52b15" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
W20260925 03:05:05.762823 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 49831.7us
I20260925 03:05:05.765188 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c23337a7cb74184a0e57b7095cb6a11" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:05.777840 24548 raft_consensus.cc:2396] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260925 03:05:05.766315 27126 raft_consensus.cc:686] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 LEADER]: Becoming Leader. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:05:05.822191 24471 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a7aa47c48404465b94a36fd819fe3c7a"
dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
mode: GRACEFUL
new_leader_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
from {username='slave'} at 127.0.0.1:38832
I20260925 03:05:05.822906 24471 raft_consensus.cc:595] T a7aa47c48404465b94a36fd819fe3c7a P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Received request to transfer leadership to TS a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:05:05.823223 24471 raft_consensus.cc:603] T a7aa47c48404465b94a36fd819fe3c7a P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260925 03:05:05.824986 27224 log.cc:924] Time spent T abd5b8dc1b7e49e3aa9af900d73ff0e7 P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.059s user 0.000s sys 0.001s
I20260925 03:05:05.829563 27126 consensus_queue.cc:237] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 5.8, Last appended by leader: 8, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:05.836519 26803 consensus_queue.cc:260] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:05.844537 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bb46bce3564841b7674e6a693bd2e6" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
W20260925 03:05:05.815565 27222 log.cc:924] Time spent T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.194s user 0.001s sys 0.000s
I20260925 03:05:05.847105 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b030d84c670b48d5bff337ccc0c77626" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:05.847831 24541 raft_consensus.cc:2454] T b030d84c670b48d5bff337ccc0c77626 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 8.
I20260925 03:05:05.860411 24514 leader_election.cc:302] T c85f5d1d3f004cf5817bb594fd8e6469 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:05.864302 26909 raft_consensus.cc:482] T 21c43af5e6844fdaacc8c625fa01fd8d P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:05.864951 26909 raft_consensus.cc:504] T 21c43af5e6844fdaacc8c625fa01fd8d P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:05.866993 26909 leader_election.cc:288] T 21c43af5e6844fdaacc8c625fa01fd8d P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:05:05.867511 26909 raft_consensus.cc:482] T 04e068a0f80941dc9138cee063f53bab P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260925 03:05:05.868057 24437 connection.cc:232] RPC call timeout handler was delayed by -2.8873218590015313s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.24740000000000004s for a response
I20260925 03:05:05.870368 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367277daf8d04778b3fa4ed94d71580d" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:05.870606 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15861cd28e354598a3c6bfc4337c34bb" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:05:05.869917 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21c43af5e6844fdaacc8c625fa01fd8d" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:05.871917 24545 raft_consensus.cc:2379] T 21c43af5e6844fdaacc8c625fa01fd8d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 in current term 5: Already voted for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in this term.
I20260925 03:05:05.885396 24471 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "edcf6e6a422d4d198d4fbb0f7d3d845d"
dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
mode: GRACEFUL
new_leader_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
from {username='slave'} at 127.0.0.1:38832
I20260925 03:05:05.886107 24471 raft_consensus.cc:595] T edcf6e6a422d4d198d4fbb0f7d3d845d P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Received request to transfer leadership to TS a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:05:05.886452 24471 raft_consensus.cc:603] T edcf6e6a422d4d198d4fbb0f7d3d845d P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260925 03:05:05.886682 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f1fac028a144058b92331c0d9b44ebf" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:05.883895 24514 leader_election.cc:302] T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:05:05.890924 24514 leader_election.cc:302] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:05:05.891776 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b627ecb4a5384ef392aa695e06599c39" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:05.908118 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be4e0aa984f4f2b81a58e382e434ad5" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:05.867951 26909 raft_consensus.cc:504] T 04e068a0f80941dc9138cee063f53bab P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:05.905156 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48ca6ccd98194cf2af7b5278d15bf4ea" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:05:05.820044 26577 raft_consensus.cc:504] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:05.887068 26803 raft_consensus.cc:3037] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43 [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:05:05.910617 24514 leader_election.cc:302] T a7aa47c48404465b94a36fd819fe3c7a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:05:05.889137 26811 raft_consensus.cc:2726] T c85f5d1d3f004cf5817bb594fd8e6469 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260925 03:05:05.918166 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "100c113a9fd84cc8adf2ec7b33ea0093" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:05.919929 24514 leader_election.cc:302] T 25102ba08e4e49388f6176badcd10a25 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:05:05.889098 27204 raft_consensus.cc:482] T 04e068a0f80941dc9138cee063f53bab P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:05:05.923503 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4ebd2b043024a548005c8b6b00b0449" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:05.904959 27190 raft_consensus.cc:2781] T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260925 03:05:05.924790 24613 raft_consensus.cc:2379] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 in current term 7: Already voted for candidate eeab93abfdb141a895002ec9f312296c in this term.
I20260925 03:05:05.924213 23300 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260925 03:05:05.914362 26937 raft_consensus.cc:2781] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260925 03:05:05.952298 23300 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
W20260925 03:05:05.946895 27228 log.cc:924] Time spent T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.132s user 0.000s sys 0.000s
I20260925 03:05:05.924154 26800 raft_consensus.cc:2781] T a7aa47c48404465b94a36fd819fe3c7a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260925 03:05:05.967016 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6b6ec010caf4689b8c90449764f5cf9" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:05.971065 23300 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260925 03:05:05.950043 26811 raft_consensus.cc:2781] T 25102ba08e4e49388f6176badcd10a25 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260925 03:05:05.925753 27190 raft_consensus.cc:482] T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:05.967041 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7aa47c48404465b94a36fd819fe3c7a" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:05.979640 24541 raft_consensus.cc:2396] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 for term 7 because replica has last-logged OpId of term: 5 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 7.
W20260925 03:05:05.968644 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 90200.6us
W20260925 03:05:05.992210 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 104010us
W20260925 03:05:05.996217 24437 connection.cc:232] RPC call timeout handler was delayed by -1.7740457560012146s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1501s for a response
W20260925 03:05:05.996640 24437 connection.cc:232] RPC call timeout handler was delayed by -1.7182457560011244s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1563s for a response
W20260925 03:05:05.996949 24437 connection.cc:232] RPC call timeout handler was delayed by -1.5913457560000097s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1704s for a response
W20260925 03:05:05.997229 24437 connection.cc:232] RPC call timeout handler was delayed by -1.5625457559999631s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1736s for a response
W20260925 03:05:05.997488 24437 connection.cc:232] RPC call timeout handler was delayed by -1.4689457559998118s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18400000000000002s for a response
W20260925 03:05:05.997759 24437 connection.cc:232] RPC call timeout handler was delayed by -1.4221457560015551s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1892s for a response
W20260925 03:05:05.998018 24437 connection.cc:232] RPC call timeout handler was delayed by -1.263745756001299s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2068s for a response
W20260925 03:05:05.998286 24437 connection.cc:232] RPC call timeout handler was delayed by -1.1593457560011302s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21840000000000004s for a response
W20260925 03:05:05.998567 24437 connection.cc:232] RPC call timeout handler was delayed by -1.1323457560010866s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2214s for a response
W20260925 03:05:05.998826 24437 connection.cc:232] RPC call timeout handler was delayed by -1.1161457560010604s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22320000000000004s for a response
W20260925 03:05:05.999092 24437 connection.cc:232] RPC call timeout handler was delayed by -1.10354575600104s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22460000000000002s for a response
I20260925 03:05:05.999738 24437 leader_election.cc:302] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:05.961894 26937 raft_consensus.cc:482] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
W20260925 03:05:06.013401 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 93560us
I20260925 03:05:06.011485 27190 raft_consensus.cc:3037] T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Advancing to term 8
I20260925 03:05:06.046949 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "420163a6cdce412ebc21c25fe3835638" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:06.047164 24514 leader_election.cc:302] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
W20260925 03:05:06.050225 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 50630.8us
W20260925 03:05:06.052903 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 478932us
W20260925 03:05:06.056444 27233 log.cc:924] Time spent T c88cbd39e01f44628a094af516be2833 P 869b0a8575b54d08ae2817fad5dd42d5: Append to log took a long time: real 0.140s user 0.000s sys 0.001s
I20260925 03:05:06.056433 27104 raft_consensus.cc:482] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:05.951234 27204 raft_consensus.cc:504] T 04e068a0f80941dc9138cee063f53bab P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:05.970638 26800 raft_consensus.cc:482] T a7aa47c48404465b94a36fd819fe3c7a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:05.984143 26811 raft_consensus.cc:482] T 25102ba08e4e49388f6176badcd10a25 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:06.066100 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5278a60f97da4e10aecd50a8d9a9a43c" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:06.048882 26937 raft_consensus.cc:3037] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:05:06.057340 27104 raft_consensus.cc:504] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:06.062238 27185 raft_consensus.cc:2726] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260925 03:05:06.081355 24588 leader_election.cc:302] T c88cbd39e01f44628a094af516be2833 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:05:06.070375 26811 raft_consensus.cc:3037] T 25102ba08e4e49388f6176badcd10a25 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:05:06.063731 26800 raft_consensus.cc:3037] T a7aa47c48404465b94a36fd819fe3c7a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:05:06.106365 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abd5b8dc1b7e49e3aa9af900d73ff0e7" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:06.107645 25198 raft_consensus.cc:1261] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43 [term 10 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
W20260925 03:05:06.104323 27234 log.cc:924] Time spent T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.142s user 0.001s sys 0.000s
I20260925 03:05:06.080507 26995 raft_consensus.cc:482] T 367277daf8d04778b3fa4ed94d71580d P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:06.071974 27218 consensus_queue.cc:1046] T 48ca6ccd98194cf2af7b5278d15bf4ea P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.039s
I20260925 03:05:06.116604 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21c43af5e6844fdaacc8c625fa01fd8d" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:06.129635 24547 raft_consensus.cc:2454] T 21c43af5e6844fdaacc8c625fa01fd8d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 5.
I20260925 03:05:06.131510 24540 raft_consensus.cc:1261] T 550118bc1a0641ce9803bdbf43f2069f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
W20260925 03:05:06.133265 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 86175.3us
I20260925 03:05:06.111481 27242 raft_consensus.cc:2781] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260925 03:05:06.138778 24514 leader_election.cc:302] T 420163a6cdce412ebc21c25fe3835638 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:06.140161 24540 raft_consensus.cc:1261] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260925 03:05:06.140532 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c85f5d1d3f004cf5817bb594fd8e6469" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:06.104545 27052 consensus_queue.cc:1035] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.028s
I20260925 03:05:06.117399 27064 raft_consensus.cc:2726] T c88cbd39e01f44628a094af516be2833 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260925 03:05:06.155624 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e8af2641e124f9589c5ccfdac785f63" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:06.156245 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e068a0f80941dc9138cee063f53bab" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:06.159755 24514 leader_election.cc:302] T 550118bc1a0641ce9803bdbf43f2069f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
W20260925 03:05:06.148130 27041 consensus_meta.cc:287] T 43366a0a2f9d4344bf4e37ffd290ef8a P eeab93abfdb141a895002ec9f312296c: Time spent flushing consensus metadata: real 0.622s user 0.011s sys 0.001s
W20260925 03:05:06.162207 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 104522us
I20260925 03:05:06.160367 24540 raft_consensus.cc:2454] T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 5.
I20260925 03:05:06.162855 24546 raft_consensus.cc:1261] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260925 03:05:06.147620 26909 leader_election.cc:288] T 04e068a0f80941dc9138cee063f53bab P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:05:06.165915 24437 leader_election.cc:302] T 37a157996e944db2906110d87ecb0c54 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:05:06.146690 27242 raft_consensus.cc:482] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:06.166417 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5332a5d329493991dff994e70ea92c" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:06.148939 26577 leader_election.cc:288] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 7 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:05:06.134636 26842 consensus_queue.cc:1035] T a7aa47c48404465b94a36fd819fe3c7a P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.030s
I20260925 03:05:06.173182 27204 leader_election.cc:288] T 04e068a0f80941dc9138cee063f53bab P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:05:06.175806 26842 consensus_queue.cc:1225] T a7aa47c48404465b94a36fd819fe3c7a P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer 869b0a8575b54d08ae2817fad5dd42d5 log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260925 03:05:06.131106 26995 raft_consensus.cc:504] T 367277daf8d04778b3fa4ed94d71580d P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
W20260925 03:05:06.177410 27239 log.cc:924] Time spent T 3d89396576884f7c8901dba54f783947 P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.117s user 0.001s sys 0.000s
I20260925 03:05:06.178889 26995 leader_election.cc:288] T 367277daf8d04778b3fa4ed94d71580d P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:06.182360 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367277daf8d04778b3fa4ed94d71580d" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:06.183053 24471 raft_consensus.cc:2454] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 6.
I20260925 03:05:06.183338 24437 leader_election.cc:302] T ceab6b0fccd54e909a64f66fe81a9a1f P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:05:06.170696 27242 raft_consensus.cc:3037] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:05:06.186525 27185 raft_consensus.cc:2781] T 37a157996e944db2906110d87ecb0c54 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260925 03:05:06.186954 27185 raft_consensus.cc:482] T 37a157996e944db2906110d87ecb0c54 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:06.187311 27185 raft_consensus.cc:3037] T 37a157996e944db2906110d87ecb0c54 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Advancing to term 8
I20260925 03:05:06.190739 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e8af2641e124f9589c5ccfdac785f63" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
W20260925 03:05:06.168481 27238 log.cc:924] Time spent T 442d4a29b1d743599f4b6b4871e26668 P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.095s user 0.001s sys 0.000s
W20260925 03:05:06.200387 27227 log.cc:924] Time spent T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5: Append to log took a long time: real 0.384s user 0.000s sys 0.006s
W20260925 03:05:06.200242 27240 log.cc:924] Time spent T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.083s user 0.001s sys 0.000s
I20260925 03:05:06.201927 27104 leader_election.cc:288] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:05:06.202189 24586 leader_election.cc:302] T 21c43af5e6844fdaacc8c625fa01fd8d P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:05:06.167330 27248 raft_consensus.cc:2726] T 420163a6cdce412ebc21c25fe3835638 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260925 03:05:06.211133 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 51451.8us
I20260925 03:05:06.167246 27250 consensus_queue.cc:1035] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.050s
I20260925 03:05:06.214794 27252 consensus_queue.cc:1035] T 550118bc1a0641ce9803bdbf43f2069f P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260925 03:05:06.216068 27104 raft_consensus.cc:2781] T ceab6b0fccd54e909a64f66fe81a9a1f P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader election won for term 7
I20260925 03:05:06.216661 27104 raft_consensus.cc:686] T ceab6b0fccd54e909a64f66fe81a9a1f P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 LEADER]: Becoming Leader. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:05:06.217051 26800 raft_consensus.cc:504] T a7aa47c48404465b94a36fd819fe3c7a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:06.217865 25198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:05:06.218482 25198 raft_consensus.cc:2454] T 7e49248643df4131a1c42e74244e05c2 P 1bc1f2e62ab44af995165bd5116a9d43 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 8.
I20260925 03:05:06.219080 26800 leader_election.cc:288] T a7aa47c48404465b94a36fd819fe3c7a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 7 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:06.219415 24541 raft_consensus.cc:1226] T a7aa47c48404465b94a36fd819fe3c7a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 6. Current term is 7. Ops: []
I20260925 03:05:06.220710 24514 leader_election.cc:302] T 11bb46bce3564841b7674e6a693bd2e6 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
W20260925 03:05:06.228927 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 45688.5us
I20260925 03:05:06.227131 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ac285175464fbda67211d9574db415" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:05:06.230579 24542 raft_consensus.cc:2379] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 in current term 7: Already voted for candidate eeab93abfdb141a895002ec9f312296c in this term.
I20260925 03:05:06.229655 24514 leader_election.cc:302] T 1414dcfe53f64a78ae413531eb8ec117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:06.212548 27022 raft_consensus.cc:2781] T 21c43af5e6844fdaacc8c625fa01fd8d P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260925 03:05:06.234911 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 76777.8us
I20260925 03:05:06.235716 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43366a0a2f9d4344bf4e37ffd290ef8a" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:06.241225 24613 raft_consensus.cc:1261] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260925 03:05:06.268312 24613 raft_consensus.cc:1261] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [term 10 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260925 03:05:06.271968 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:06.274217 24437 leader_election.cc:302] T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:05:06.224574 26800 raft_consensus.cc:2726] T 550118bc1a0641ce9803bdbf43f2069f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260925 03:05:06.278579 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 48988.2us
I20260925 03:05:06.279521 24514 leader_election.cc:302] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:06.284255 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:06.284917 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e068a0f80941dc9138cee063f53bab" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:06.286569 24542 raft_consensus.cc:2396] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 for term 9 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20260925 03:05:06.235668 27248 raft_consensus.cc:2781] T 11bb46bce3564841b7674e6a693bd2e6 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260925 03:05:06.235571 27022 raft_consensus.cc:482] T 21c43af5e6844fdaacc8c625fa01fd8d P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:05:06.291308 24613 raft_consensus.cc:2379] T 04e068a0f80941dc9138cee063f53bab P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1bc1f2e62ab44af995165bd5116a9d43 in current term 5: Already voted for candidate eeab93abfdb141a895002ec9f312296c in this term.
I20260925 03:05:06.291651 24586 leader_election.cc:302] T 04e068a0f80941dc9138cee063f53bab P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:05:06.296036 24437 leader_election.cc:302] T 7993f9c3e2a44f408bbf5d1e395e2306 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:05:06.228222 27252 consensus_queue.cc:1035] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.065s
I20260925 03:05:06.303712 25198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e068a0f80941dc9138cee063f53bab" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:05:06.304487 25198 raft_consensus.cc:2454] T 04e068a0f80941dc9138cee063f53bab P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 4.
I20260925 03:05:06.306479 27077 raft_consensus.cc:2781] T 7993f9c3e2a44f408bbf5d1e395e2306 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260925 03:05:06.306995 27077 raft_consensus.cc:482] T 7993f9c3e2a44f408bbf5d1e395e2306 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:06.307350 27077 raft_consensus.cc:3037] T 7993f9c3e2a44f408bbf5d1e395e2306 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:05:06.309499 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367277daf8d04778b3fa4ed94d71580d" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:06.286154 26946 raft_consensus.cc:2781] T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260925 03:05:06.313054 26946 raft_consensus.cc:482] T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:06.313470 26946 raft_consensus.cc:3037] T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:05:06.315354 27250 consensus_queue.cc:1035] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260925 03:05:06.317286 27250 consensus_queue.cc:1035] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260925 03:05:06.317957 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcf6e6a422d4d198d4fbb0f7d3d845d" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:06.319186 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e8af2641e124f9589c5ccfdac785f63" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:05:06.320952 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e725b82985746b39c0cc8f671c95353" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:06.321434 27077 raft_consensus.cc:504] T 7993f9c3e2a44f408bbf5d1e395e2306 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:06.323906 27077 leader_election.cc:288] T 7993f9c3e2a44f408bbf5d1e395e2306 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:06.292797 27126 raft_consensus.cc:2726] T 1414dcfe53f64a78ae413531eb8ec117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260925 03:05:06.326231 27126 raft_consensus.cc:2726] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260925 03:05:06.326793 27126 raft_consensus.cc:482] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:06.327231 27126 raft_consensus.cc:504] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:06.329854 27190 raft_consensus.cc:504] T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:06.331909 27190 leader_election.cc:288] T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 8 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:06.333856 26811 raft_consensus.cc:504] T 25102ba08e4e49388f6176badcd10a25 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:06.335887 26811 leader_election.cc:288] T 25102ba08e4e49388f6176badcd10a25 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 6 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:06.337333 24437 leader_election.cc:302] T 7b8e0d260a884ecdb20f79de46c36b19 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:05:06.338647 24437 leader_election.cc:302] T 1414dcfe53f64a78ae413531eb8ec117 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:06.339429 27249 raft_consensus.cc:2726] T 1414dcfe53f64a78ae413531eb8ec117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260925 03:05:06.339814 24437 leader_election.cc:302] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:05:06.338637 27077 raft_consensus.cc:2781] T 7b8e0d260a884ecdb20f79de46c36b19 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260925 03:05:06.340648 27249 raft_consensus.cc:2781] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader election won for term 7
W20260925 03:05:06.341291 27251 log.cc:924] Time spent T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.068s user 0.000s sys 0.000s
I20260925 03:05:06.341408 27249 raft_consensus.cc:686] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 LEADER]: Becoming Leader. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:05:06.342305 24548 raft_consensus.cc:1226] T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 7. Current term is 8. Ops: []
I20260925 03:05:06.344594 24514 leader_election.cc:302] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:05:06.293702 27254 raft_consensus.cc:482] T e6b6ec010caf4689b8c90449764f5cf9 P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:06.347244 24514 leader_election.cc:302] T 442d4a29b1d743599f4b6b4871e26668 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:05:06.348479 24437 leader_election.cc:398] T 3d89396576884f7c8901dba54f783947 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: Vote denied by peer a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389) with higher term. Message: Invalid argument: T 3d89396576884f7c8901dba54f783947 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 4. Current term is 5.
W20260925 03:05:06.348855 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 46801.6us
I20260925 03:05:06.294895 27248 raft_consensus.cc:482] T 11bb46bce3564841b7674e6a693bd2e6 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:06.299081 27022 raft_consensus.cc:3037] T 21c43af5e6844fdaacc8c625fa01fd8d P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:05:06.358527 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7691c6b475428c82d6181d693d26f3" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:06.359947 24470 raft_consensus.cc:1261] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260925 03:05:06.226347 27104 consensus_queue.cc:237] T ceab6b0fccd54e909a64f66fe81a9a1f P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:06.352200 24514 leader_election.cc:398] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 6 election: Vote denied by peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187) with higher term. Message: Invalid argument: T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 6. Current term is 7.
I20260925 03:05:06.363517 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "550118bc1a0641ce9803bdbf43f2069f" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:06.367372 24514 leader_election.cc:401] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
W20260925 03:05:06.364266 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 72720.2us
I20260925 03:05:06.355998 24511 leader_election.cc:302] T edcf6e6a422d4d198d4fbb0f7d3d845d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:06.366039 24437 leader_election.cc:302] T ac9357d144aa49789faaf31fe5e6bf89 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:06.366900 24470 raft_consensus.cc:1261] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260925 03:05:06.370316 24511 leader_election.cc:302] T c88cbd39e01f44628a094af516be2833 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:05:06.370553 24465 raft_consensus.cc:1261] T 25102ba08e4e49388f6176badcd10a25 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260925 03:05:06.351804 26800 raft_consensus.cc:2781] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260925 03:05:06.345506 27077 raft_consensus.cc:482] T 7b8e0d260a884ecdb20f79de46c36b19 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:06.374845 26800 raft_consensus.cc:482] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:06.375069 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ac285175464fbda67211d9574db415" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:06.375384 27077 raft_consensus.cc:3037] T 7b8e0d260a884ecdb20f79de46c36b19 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Advancing to term 7
W20260925 03:05:06.377020 27235 log.cc:924] Time spent T 71f461a6e2ff434993b78a2d077f0abd P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.552s user 0.000s sys 0.007s
I20260925 03:05:06.377638 27126 leader_election.cc:288] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:06.378298 27126 raft_consensus.cc:2781] T c88cbd39e01f44628a094af516be2833 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260925 03:05:06.378665 27126 raft_consensus.cc:482] T c88cbd39e01f44628a094af516be2833 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:05:06.379017 27126 raft_consensus.cc:3037] T c88cbd39e01f44628a094af516be2833 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Advancing to term 8
I20260925 03:05:06.380955 26800 raft_consensus.cc:3037] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:05:06.359316 27248 raft_consensus.cc:3037] T 11bb46bce3564841b7674e6a693bd2e6 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:05:06.360134 27249 consensus_queue.cc:237] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:06.357123 26811 raft_consensus.cc:2781] T 442d4a29b1d743599f4b6b4871e26668 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260925 03:05:06.396652 24466 raft_consensus.cc:2379] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c in current term 7: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:05:06.404914 24540 raft_consensus.cc:1261] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 4. (term mismatch)
I20260925 03:05:06.410537 24540 pending_rounds.cc:76] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Aborting all ops after (but not including) 3
I20260925 03:05:06.411566 24471 raft_consensus.cc:2454] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 8.
I20260925 03:05:06.410746 24540 pending_rounds.cc:97] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Aborting uncommitted NO_OP operation due to leader change: 4.4
I20260925 03:05:06.425191 24540 raft_consensus.cc:2866] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260925 03:05:06.425578 24540 pending_rounds.cc:97] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Aborting uncommitted NO_OP operation due to leader change: 6.5
I20260925 03:05:06.425793 24540 raft_consensus.cc:2866] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260925 03:05:06.430979 24540 raft_consensus.cc:1226] T 25102ba08e4e49388f6176badcd10a25 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 5. Current term is 6. Ops: []
I20260925 03:05:06.431282 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06f4a7eaf6534e4986c9fd35911038e4" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:06.433732 24616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7aa47c48404465b94a36fd819fe3c7a" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:05:06.434010 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c23337a7cb74184a0e57b7095cb6a11" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:05:06.435002 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:06.382707 27242 raft_consensus.cc:504] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:06.437098 24620 raft_consensus.cc:1226] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 6. Current term is 7. Ops: [6.6-6.6]
I20260925 03:05:06.369479 26970 raft_consensus.cc:2781] T 04e068a0f80941dc9138cee063f53bab P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260925 03:05:06.435696 24613 raft_consensus.cc:2454] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 8.
I20260925 03:05:06.441977 26970 raft_consensus.cc:482] T 04e068a0f80941dc9138cee063f53bab P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:05:06.442248 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5f9b503b0ab43999d582015a367441e" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:05:06.442925 24622 raft_consensus.cc:3037] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Advancing to term 9
I20260925 03:05:06.443218 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7691c6b475428c82d6181d693d26f3" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:06.443980 24617 raft_consensus.cc:2379] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 8: Already voted for candidate eeab93abfdb141a895002ec9f312296c in this term.
I20260925 03:05:06.445173 26970 raft_consensus.cc:3037] T 04e068a0f80941dc9138cee063f53bab P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:05:06.445295 24618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25102ba08e4e49388f6176badcd10a25" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:05:06.454298 26937 raft_consensus.cc:504] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:06.455793 24542 raft_consensus.cc:2379] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 in current term 7: Already voted for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in this term.
I20260925 03:05:06.455680 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "100c113a9fd84cc8adf2ec7b33ea0093" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:06.444977 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c81a1b3b616492ab6c964f45af36077" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
W20260925 03:05:06.428424 27256 log.cc:924] Time spent T 00000000000000000000000000000000 P 2d92cc0732c4490eb7d98d353826cce1: Append to log took a long time: real 0.138s user 0.002s sys 0.000s
I20260925 03:05:06.404445 27255 raft_consensus.cc:2726] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187) with higher term. Message: Invalid argument: T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 6. Current term is 7.
I20260925 03:05:06.468132 24470 raft_consensus.cc:1261] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260925 03:05:06.423097 26811 raft_consensus.cc:482] T 442d4a29b1d743599f4b6b4871e26668 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:06.444921 24615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367277daf8d04778b3fa4ed94d71580d" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:06.404826 27190 raft_consensus.cc:2726] T edcf6e6a422d4d198d4fbb0f7d3d845d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260925 03:05:06.471915 26741 raft_consensus.cc:3032] T 48ca6ccd98194cf2af7b5278d15bf4ea P eeab93abfdb141a895002ec9f312296c [term 4 LEADER]: Stepping down as leader of term 4
I20260925 03:05:06.394533 27230 consensus_queue.cc:1035] T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.083s
I20260925 03:05:06.477401 27146 consensus_queue.cc:1035] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260925 03:05:06.487145 26970 raft_consensus.cc:504] T 04e068a0f80941dc9138cee063f53bab P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:06.345520 27260 consensus_queue.cc:1046] T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 5.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.128s
I20260925 03:05:06.497946 24545 raft_consensus.cc:1226] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 6. Current term is 7. Ops: []
I20260925 03:05:06.498487 24466 raft_consensus.cc:1261] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260925 03:05:06.359701 27254 raft_consensus.cc:504] T e6b6ec010caf4689b8c90449764f5cf9 P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:06.395278 27040 consensus_queue.cc:1046] T a7aa47c48404465b94a36fd819fe3c7a P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.094s
I20260925 03:05:06.493858 26811 raft_consensus.cc:3037] T 442d4a29b1d743599f4b6b4871e26668 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Advancing to term 8
I20260925 03:05:06.495270 26741 raft_consensus.cc:729] T 48ca6ccd98194cf2af7b5278d15bf4ea P eeab93abfdb141a895002ec9f312296c [term 4 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:05:06.515450 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21c43af5e6844fdaacc8c625fa01fd8d" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:06.516191 24466 raft_consensus.cc:2454] T 21c43af5e6844fdaacc8c625fa01fd8d P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 5.
I20260925 03:05:06.495404 26946 raft_consensus.cc:504] T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:06.509056 26842 raft_consensus.cc:482] T 5278a60f97da4e10aecd50a8d9a9a43c P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:06.495177 27077 raft_consensus.cc:504] T 7b8e0d260a884ecdb20f79de46c36b19 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:05:06.531072 24511 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 160822us
I20260925 03:05:06.532579 24542 raft_consensus.cc:1226] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 6. Current term is 7. Ops: []
I20260925 03:05:06.529479 27271 raft_consensus.cc:482] T 3d89396576884f7c8901dba54f783947 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:06.388636 26800 raft_consensus.cc:504] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:06.528950 27265 raft_consensus.cc:2726] T ac9357d144aa49789faaf31fe5e6bf89 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260925 03:05:06.539309 24470 raft_consensus.cc:1226] T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 5. Current term is 6. Ops: []
I20260925 03:05:06.532599 24511 leader_election.cc:398] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 6 election: Vote denied by peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105) with higher term. Message: Invalid argument: T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 6. Current term is 7.
W20260925 03:05:06.540966 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 175020us
I20260925 03:05:06.552452 24542 raft_consensus.cc:1261] T 8e5819c15eac4544a57e5ee6c9d4f117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
W20260925 03:05:06.553223 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 201169us
W20260925 03:05:06.525492 24436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 210514us
I20260925 03:05:06.529798 27009 raft_consensus.cc:482] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:06.556567 24514 leader_election.cc:302] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:05:06.556807 27009 raft_consensus.cc:504] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:06.561434 27248 raft_consensus.cc:504] T 11bb46bce3564841b7674e6a693bd2e6 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:06.563043 27009 leader_election.cc:288] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:05:06.534988 26842 raft_consensus.cc:504] T 5278a60f97da4e10aecd50a8d9a9a43c P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:06.566588 27255 raft_consensus.cc:2781] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260925 03:05:06.567087 27185 raft_consensus.cc:504] T 37a157996e944db2906110d87ecb0c54 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:06.567705 26842 leader_election.cc:288] T 5278a60f97da4e10aecd50a8d9a9a43c P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:05:06.568912 24511 leader_election.cc:302] T ac9357d144aa49789faaf31fe5e6bf89 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:06.570137 24511 leader_election.cc:302] T 8ac466793d094325ace224708b2158ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 1bc1f2e62ab44af995165bd5116a9d43, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:06.571292 24467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e8af2641e124f9589c5ccfdac785f63" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:06.567389 27022 raft_consensus.cc:504] T 21c43af5e6844fdaacc8c625fa01fd8d P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:06.578892 24467 raft_consensus.cc:2379] T 9e8af2641e124f9589c5ccfdac785f63 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c in current term 7: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:05:06.579981 24546 raft_consensus.cc:1226] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 6. Current term is 7. Ops: []
I20260925 03:05:06.580507 24467 pending_rounds.cc:76] T a7aa47c48404465b94a36fd819fe3c7a P 869b0a8575b54d08ae2817fad5dd42d5: Aborting all ops after (but not including) 1
I20260925 03:05:06.580968 24467 pending_rounds.cc:97] T a7aa47c48404465b94a36fd819fe3c7a P 869b0a8575b54d08ae2817fad5dd42d5: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20260925 03:05:06.581243 24467 raft_consensus.cc:2866] T a7aa47c48404465b94a36fd819fe3c7a P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260925 03:05:06.579172 24472 raft_consensus.cc:1226] T 7b8e0d260a884ecdb20f79de46c36b19 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 6. Current term is 7. Ops: []
I20260925 03:05:06.543946 27230 raft_consensus.cc:482] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:06.588423 24469 raft_consensus.cc:1261] T 442d4a29b1d743599f4b6b4871e26668 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260925 03:05:06.588577 24514 leader_election.cc:302] T 367277daf8d04778b3fa4ed94d71580d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:05:06.567009 27255 raft_consensus.cc:482] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:06.601323 24437 leader_election.cc:302] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:06.605110 24618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7993f9c3e2a44f408bbf5d1e395e2306" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:05:06.598332 24514 leader_election.cc:302] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:06.641408 24511 leader_election.cc:302] T cc2df40db08649c9a75e90000b878559 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:06.558892 26937 leader_election.cc:288] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 7 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:06.657287 24469 raft_consensus.cc:1261] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260925 03:05:06.587757 27260 raft_consensus.cc:2726] T ac9357d144aa49789faaf31fe5e6bf89 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260925 03:05:06.639293 27255 raft_consensus.cc:3037] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Advancing to term 9
I20260925 03:05:06.639609 27269 raft_consensus.cc:2726] T 367277daf8d04778b3fa4ed94d71580d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260925 03:05:06.675271 24622 raft_consensus.cc:2454] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 9.
I20260925 03:05:06.584118 27052 consensus_queue.cc:1035] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.077s
I20260925 03:05:06.677971 24619 raft_consensus.cc:1226] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 8. Current term is 9. Ops: [8.5-8.5]
W20260925 03:05:06.686209 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 84998.7us
I20260925 03:05:06.686990 24437 leader_election.cc:302] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:06.687978 24469 raft_consensus.cc:1261] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260925 03:05:06.688311 24437 leader_election.cc:302] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:06.689039 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceab6b0fccd54e909a64f66fe81a9a1f" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:06.690536 24469 raft_consensus.cc:1226] T 37a157996e944db2906110d87ecb0c54 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 7. Current term is 8. Ops: []
I20260925 03:05:06.554605 26741 consensus_queue.cc:260] T 48ca6ccd98194cf2af7b5278d15bf4ea P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:06.606703 27230 raft_consensus.cc:504] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:06.557405 27271 raft_consensus.cc:504] T 3d89396576884f7c8901dba54f783947 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:06.680110 27242 leader_election.cc:288] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 7 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:06.658249 27252 consensus_queue.cc:1046] T 25102ba08e4e49388f6176badcd10a25 P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.039s
I20260925 03:05:06.677861 27260 raft_consensus.cc:2726] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260925 03:05:06.710275 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5278a60f97da4e10aecd50a8d9a9a43c" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:06.735577 24471 raft_consensus.cc:2454] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 5.
W20260925 03:05:06.714313 24436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.4:43979, user_credentials={real_user=slave}} blocked reactor thread for 130946us
I20260925 03:05:06.708370 24437 leader_election.cc:302] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:05:06.671309 26756 consensus_queue.cc:1046] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.039s
I20260925 03:05:06.697782 27269 raft_consensus.cc:2726] T cc2df40db08649c9a75e90000b878559 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260925 03:05:06.740661 24464 raft_consensus.cc:1261] T 3d89396576884f7c8901dba54f783947 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260925 03:05:06.740967 24437 leader_election.cc:302] T 11bb46bce3564841b7674e6a693bd2e6 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:06.714493 27248 leader_election.cc:288] T 11bb46bce3564841b7674e6a693bd2e6 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:06.709888 26970 leader_election.cc:288] T 04e068a0f80941dc9138cee063f53bab P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 6 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:05:06.677987 27071 consensus_queue.cc:1046] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.039s
I20260925 03:05:06.752789 27268 raft_consensus.cc:2726] T 8ac466793d094325ace224708b2158ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260925 03:05:06.738651 27254 leader_election.cc:288] T e6b6ec010caf4689b8c90449764f5cf9 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:05:06.754073 27271 leader_election.cc:288] T 3d89396576884f7c8901dba54f783947 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
W20260925 03:05:06.753763 27276 log.cc:924] Time spent T 8e5819c15eac4544a57e5ee6c9d4f117 P 869b0a8575b54d08ae2817fad5dd42d5: Append to log took a long time: real 0.053s user 0.000s sys 0.000s
W20260925 03:05:06.787586 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 46890.9us
I20260925 03:05:06.744983 27146 raft_consensus.cc:3032] T 25102ba08e4e49388f6176badcd10a25 P eeab93abfdb141a895002ec9f312296c [term 5 LEADER]: Stepping down as leader of term 5
I20260925 03:05:06.788535 24437 leader_election.cc:302] T 21c43af5e6844fdaacc8c625fa01fd8d P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
W20260925 03:05:06.789762 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 102872us
W20260925 03:05:06.796281 27270 log.cc:924] Time spent T 550118bc1a0641ce9803bdbf43f2069f P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.089s user 0.000s sys 0.000s
I20260925 03:05:06.792435 27146 raft_consensus.cc:729] T 25102ba08e4e49388f6176badcd10a25 P eeab93abfdb141a895002ec9f312296c [term 5 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:05:06.763228 27252 raft_consensus.cc:3032] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [term 7 LEADER]: Stepping down as leader of term 7
I20260925 03:05:06.806734 27252 raft_consensus.cc:729] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [term 7 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:05:06.807688 27252 consensus_queue.cc:260] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:06.809115 27252 raft_consensus.cc:3037] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Advancing to term 8
I20260925 03:05:06.809859 27190 raft_consensus.cc:3032] T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 LEADER]: Stepping down as leader of term 6
I20260925 03:05:06.810396 27190 raft_consensus.cc:729] T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:05:06.811278 27190 consensus_queue.cc:260] T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 4, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:06.812402 27190 raft_consensus.cc:3037] T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Advancing to term 7
W20260925 03:05:06.725135 27009 consensus_queue.cc:1159] T 48ca6ccd98194cf2af7b5278d15bf4ea P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" responder_term: 4 status { last_received { term: 4 index: 3 } last_committed_idx: 3 last_received_current_leader { term: 4 index: 3 } }
I20260925 03:05:06.792672 26800 leader_election.cc:288] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 7 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:06.758059 26946 leader_election.cc:288] T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:06.764680 27077 leader_election.cc:288] T 7b8e0d260a884ecdb20f79de46c36b19 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:06.815063 27185 leader_election.cc:288] T 37a157996e944db2906110d87ecb0c54 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 8 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:06.758641 26937 raft_consensus.cc:482] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:06.816045 26937 raft_consensus.cc:504] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:06.818427 26937 leader_election.cc:288] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
W20260925 03:05:06.823179 27266 log.cc:924] Time spent T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43: Append to log took a long time: real 0.107s user 0.001s sys 0.000s
I20260925 03:05:06.824661 27230 leader_election.cc:288] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:05:06.816715 27226 raft_consensus.cc:3032] T a7aa47c48404465b94a36fd819fe3c7a P eeab93abfdb141a895002ec9f312296c [term 6 LEADER]: Stepping down as leader of term 6
I20260925 03:05:06.762513 24468 raft_consensus.cc:2454] T 550118bc1a0641ce9803bdbf43f2069f P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 7.
I20260925 03:05:06.828931 24466 raft_consensus.cc:1261] T 550118bc1a0641ce9803bdbf43f2069f P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260925 03:05:06.741369 27262 consensus_queue.cc:1035] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.049s
I20260925 03:05:06.836126 25197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e068a0f80941dc9138cee063f53bab" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43"
I20260925 03:05:06.836796 25197 raft_consensus.cc:3037] T 04e068a0f80941dc9138cee063f53bab P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Advancing to term 6
I20260925 03:05:06.750453 26900 consensus_queue.cc:1035] T 25102ba08e4e49388f6176badcd10a25 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.076s
I20260925 03:05:06.844951 27287 raft_consensus.cc:2781] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260925 03:05:06.845575 27287 raft_consensus.cc:482] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:06.845954 27287 raft_consensus.cc:3037] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Advancing to term 9
I20260925 03:05:06.848779 26741 raft_consensus.cc:3037] T 48ca6ccd98194cf2af7b5278d15bf4ea P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:05:06.844991 27294 raft_consensus.cc:2726] T 11bb46bce3564841b7674e6a693bd2e6 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260925 03:05:06.853878 27287 raft_consensus.cc:504] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:06.856004 27287 leader_election.cc:288] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 9 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:06.844954 25197 raft_consensus.cc:2454] T 04e068a0f80941dc9138cee063f53bab P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 6.
I20260925 03:05:06.770120 27273 raft_consensus.cc:2726] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260925 03:05:06.858349 26811 raft_consensus.cc:504] T 442d4a29b1d743599f4b6b4871e26668 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:06.833705 27022 leader_election.cc:288] T 21c43af5e6844fdaacc8c625fa01fd8d P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 6 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
W20260925 03:05:06.870715 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 77382.9us
I20260925 03:05:06.825886 27226 raft_consensus.cc:729] T a7aa47c48404465b94a36fd819fe3c7a P eeab93abfdb141a895002ec9f312296c [term 6 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
W20260925 03:05:06.879441 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 91003.9us
W20260925 03:05:06.876097 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 162815us
I20260925 03:05:06.883522 24437 leader_election.cc:302] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: 1bc1f2e62ab44af995165bd5116a9d43, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:05:06.883443 24588 leader_election.cc:302] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5
W20260925 03:05:06.874783 27264 log.cc:924] Time spent T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.216s user 0.003s sys 0.000s
I20260925 03:05:06.864619 27053 raft_consensus.cc:482] T a4abe0ea757e4886be4945c5001e529a P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:06.891623 27293 raft_consensus.cc:2726] T 21c43af5e6844fdaacc8c625fa01fd8d P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260925 03:05:06.900291 26762 consensus_queue.cc:1046] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20260925 03:05:06.891745 27277 log.cc:924] Time spent T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.087s user 0.001s sys 0.000s
I20260925 03:05:06.892403 27253 raft_consensus.cc:2781] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Leader election won for term 8
I20260925 03:05:06.904985 26970 consensus_queue.cc:1035] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260925 03:05:06.905375 24588 leader_election.cc:302] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:05:06.905894 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6b6ec010caf4689b8c90449764f5cf9" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:06.930775 24466 raft_consensus.cc:2454] T e6b6ec010caf4689b8c90449764f5cf9 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 6.
I20260925 03:05:06.931995 27255 raft_consensus.cc:504] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:06.919611 27280 raft_consensus.cc:2726] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260925 03:05:06.860661 26811 leader_election.cc:288] T 442d4a29b1d743599f4b6b4871e26668 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 8 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:06.920594 26842 raft_consensus.cc:3032] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [term 6 LEADER]: Stepping down as leader of term 6
I20260925 03:05:06.862721 27146 consensus_queue.cc:260] T 25102ba08e4e49388f6176badcd10a25 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:06.919618 27053 raft_consensus.cc:504] T a4abe0ea757e4886be4945c5001e529a P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:06.955447 24544 raft_consensus.cc:1226] T 11bb46bce3564841b7674e6a693bd2e6 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 3. Current term is 4. Ops: []
I20260925 03:05:06.954988 24547 raft_consensus.cc:1226] T a7aa47c48404465b94a36fd819fe3c7a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 6. Current term is 7. Ops: []
I20260925 03:05:06.907346 24437 leader_election.cc:302] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:06.940922 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceab6b0fccd54e909a64f66fe81a9a1f" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:06.956928 24546 raft_consensus.cc:2454] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 7.
I20260925 03:05:06.943171 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5278a60f97da4e10aecd50a8d9a9a43c" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:06.957945 24545 raft_consensus.cc:2454] T 5278a60f97da4e10aecd50a8d9a9a43c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 4.
W20260925 03:05:06.947192 24511 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 305887us
W20260925 03:05:06.949000 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 350729us
I20260925 03:05:06.960073 24514 leader_election.cc:302] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:05:06.952411 24542 raft_consensus.cc:1226] T 442d4a29b1d743599f4b6b4871e26668 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 7. Current term is 8. Ops: []
I20260925 03:05:06.933001 27022 raft_consensus.cc:3032] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P eeab93abfdb141a895002ec9f312296c [term 6 LEADER]: Stepping down as leader of term 6
I20260925 03:05:06.933565 25165 leader_election.cc:302] T e6b6ec010caf4689b8c90449764f5cf9 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5; no voters:
I20260925 03:05:06.954440 26741 raft_consensus.cc:2726] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260925 03:05:06.969715 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 62458.2us
I20260925 03:05:06.970556 24437 leader_election.cc:302] T b627ecb4a5384ef392aa695e06599c39 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:06.972034 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e068a0f80941dc9138cee063f53bab" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:05:06.972718 24543 raft_consensus.cc:3037] T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:05:06.948458 26842 raft_consensus.cc:729] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [term 6 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:05:07.003839 24514 leader_election.cc:398] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 election: Vote denied by peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187) with higher term. Message: Invalid argument: T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 5. Current term is 6.
I20260925 03:05:07.005290 24437 leader_election.cc:302] T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:07.007287 24514 leader_election.cc:302] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:07.006465 24588 leader_election.cc:302] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:05:06.964259 27022 raft_consensus.cc:729] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P eeab93abfdb141a895002ec9f312296c [term 6 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:05:07.022743 24514 leader_election.cc:302] T 43366a0a2f9d4344bf4e37ffd290ef8a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:07.023639 24437 leader_election.cc:302] T 100c113a9fd84cc8adf2ec7b33ea0093 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:06.969149 27254 raft_consensus.cc:2781] T e6b6ec010caf4689b8c90449764f5cf9 P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260925 03:05:07.029680 27280 raft_consensus.cc:2726] T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260925 03:05:07.030174 24539 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33068 (request call id 186) took 1204 ms. Trace:
I20260925 03:05:07.030580 24539 rpcz_store.cc:276] 0925 03:05:05.825590 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:05:05.825712 (+ 122us) service_pool.cc:227] Handling call
0925 03:05:05.842871 (+ 17159us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:05:05.847875 (+ 5004us) raft_consensus.cc:1520] Early marking committed up to index 7
0925 03:05:05.879702 (+ 31827us) log.cc:842] Serialized 14 byte log entry
0925 03:05:06.254171 (+374469us) spinlock_profiling.cc:243] Waited 1.52 ms on lock 0x7b18003282a0. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f437c4478b1 00007f437c454661 00007f437358a345 00007f437c3c1a39 00007f43735a78bb 00007f43735a5cbf 00007f43735a7e22 00007f43761515bb 00007f4376157063 00007f437615757b 00007f43764da88a
0925 03:05:06.554893 (+300722us) log.cc:842] Serialized 14 byte log entry
0925 03:05:06.555072 (+ 179us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:05:06.580463 (+ 25391us) log.cc:842] Serialized 26 byte log entry
0925 03:05:06.580626 (+ 163us) raft_consensus.cc:1631] Marking committed up to 7
0925 03:05:06.580659 (+ 33us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:05:06.581142 (+ 483us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:05:06.818974 (+237832us) spinlock_profiling.cc:243] Waited 6.82 ms on lock 0x7b44000232cc. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f43733ecf11 00007f43734baa95 00007f43734bb4b2 00007f4376163511 00007f437615ee1a 00007f4376152bee 00007f4376151f36 00007f4376151b58 00007f437615b551 00007f437615b509 00007f437615b499
0925 03:05:07.029972 (+210998us) raft_consensus.cc:1671] finished
0925 03:05:07.029993 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:05:07.030114 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10166,"prepare.queue_time_us":135885,"prepare.run_cpu_time_us":3289,"prepare.run_wall_time_us":19128,"raft.queue_time_us":13636,"raft.run_cpu_time_us":292,"raft.run_wall_time_us":34705,"spinlock_wait_cycles":19178112,"thread_start_us":299187,"threads_started":2,"wal-append.queue_time_us":630556}
I20260925 03:05:06.963817 27262 consensus_queue.cc:1035] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.061s
W20260925 03:05:07.026981 24511 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 64085.2us
I20260925 03:05:07.027267 24437 leader_election.cc:302] T b4ebd2b043024a548005c8b6b00b0449 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: 1bc1f2e62ab44af995165bd5116a9d43, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:05:07.037634 24514 leader_election.cc:302] T 21c43af5e6844fdaacc8c625fa01fd8d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:05:07.038059 24437 leader_election.cc:302] T 0be4e0aa984f4f2b81a58e382e434ad5 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:07.008184 27249 raft_consensus.cc:2726] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260925 03:05:06.954993 26577 consensus_queue.cc:1046] T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.055s
I20260925 03:05:07.009938 27291 consensus_queue.cc:1035] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.023s
I20260925 03:05:07.010214 27185 raft_consensus.cc:2726] T b627ecb4a5384ef392aa695e06599c39 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260925 03:05:07.039067 24585 leader_election.cc:302] T 04e068a0f80941dc9138cee063f53bab P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:05:07.040042 24437 leader_election.cc:302] T e6b6ec010caf4689b8c90449764f5cf9 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: 1bc1f2e62ab44af995165bd5116a9d43, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:05:06.948046 27226 consensus_queue.cc:260] T a7aa47c48404465b94a36fd819fe3c7a P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:06.973548 26811 consensus_queue.cc:1035] T a4abe0ea757e4886be4945c5001e529a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.055s
I20260925 03:05:07.009451 27190 raft_consensus.cc:2781] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260925 03:05:07.041702 24511 leader_election.cc:302] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:07.024744 27279 raft_consensus.cc:2726] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260925 03:05:07.046150 24437 leader_election.cc:302] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:07.028736 27254 raft_consensus.cc:482] T e6b6ec010caf4689b8c90449764f5cf9 P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:05:07.028939 27293 raft_consensus.cc:2726] T 100c113a9fd84cc8adf2ec7b33ea0093 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260925 03:05:07.045991 24514 leader_election.cc:302] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:05:07.050827 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35992316cb8d4c028333388eff02ecde" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:07.100147 24539 raft_consensus.cc:2454] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 7.
I20260925 03:05:07.052307 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21c43af5e6844fdaacc8c625fa01fd8d" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:05:07.038607 27253 raft_consensus.cc:686] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [term 8 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
W20260925 03:05:07.053730 27285 log.cc:924] Time spent T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5: Append to log took a long time: real 0.152s user 0.002s sys 0.000s
I20260925 03:05:07.054080 26577 raft_consensus.cc:3032] T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c [term 6 LEADER]: Stepping down as leader of term 6
I20260925 03:05:06.972370 27273 consensus_queue.cc:1046] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.069s
I20260925 03:05:07.057051 27053 leader_election.cc:288] T a4abe0ea757e4886be4945c5001e529a P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:05:07.057260 26762 consensus_queue.cc:1046] T 74443a0a1dc34d88a2f42e3b722db405 P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260925 03:05:07.057931 27241 consensus_queue.cc:1035] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260925 03:05:07.056264 27041 consensus_queue.cc:1046] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.002s
I20260925 03:05:07.038930 27282 raft_consensus.cc:2726] T b4ebd2b043024a548005c8b6b00b0449 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260925 03:05:07.095422 26800 raft_consensus.cc:2726] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260925 03:05:07.056951 27230 consensus_queue.cc:1035] T 442d4a29b1d743599f4b6b4871e26668 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.002s
I20260925 03:05:07.098228 27254 raft_consensus.cc:3037] T e6b6ec010caf4689b8c90449764f5cf9 P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:05:07.102270 26946 raft_consensus.cc:2726] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260925 03:05:07.118427 27104 raft_consensus.cc:3032] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 LEADER]: Stepping down as leader of term 8
I20260925 03:05:07.118839 27104 raft_consensus.cc:729] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:05:07.119369 27041 raft_consensus.cc:3032] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [term 6 LEADER]: Stepping down as leader of term 6
I20260925 03:05:07.120072 27280 raft_consensus.cc:482] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1bc1f2e62ab44af995165bd5116a9d43)
I20260925 03:05:07.120931 27267 raft_consensus.cc:2726] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260925 03:05:07.120613 27280 raft_consensus.cc:504] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:07.122802 27280 leader_election.cc:288] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:05:07.059360 27262 raft_consensus.cc:2781] T 04e068a0f80941dc9138cee063f53bab P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Leader election won for term 6
I20260925 03:05:07.123502 27262 raft_consensus.cc:686] T 04e068a0f80941dc9138cee063f53bab P eeab93abfdb141a895002ec9f312296c [term 6 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:05:07.124456 27262 consensus_queue.cc:237] T 04e068a0f80941dc9138cee063f53bab P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:07.059845 27291 consensus_queue.cc:1046] T 37a157996e944db2906110d87ecb0c54 P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20260925 03:05:07.058902 24585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.4:43979, user_credentials={real_user=slave}} blocked reactor thread for 185201us
I20260925 03:05:07.081339 27190 raft_consensus.cc:482] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:07.128342 27190 raft_consensus.cc:3037] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Advancing to term 8
I20260925 03:05:07.128407 27291 raft_consensus.cc:3032] T 37a157996e944db2906110d87ecb0c54 P eeab93abfdb141a895002ec9f312296c [term 7 LEADER]: Stepping down as leader of term 7
I20260925 03:05:07.128923 27291 raft_consensus.cc:729] T 37a157996e944db2906110d87ecb0c54 P eeab93abfdb141a895002ec9f312296c [term 7 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:05:07.129701 27291 consensus_queue.cc:260] T 37a157996e944db2906110d87ecb0c54 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:07.130910 27291 raft_consensus.cc:3037] T 37a157996e944db2906110d87ecb0c54 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Advancing to term 8
I20260925 03:05:07.060353 27272 raft_consensus.cc:2726] T 0be4e0aa984f4f2b81a58e382e434ad5 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260925 03:05:07.131469 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b8e0d260a884ecdb20f79de46c36b19" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:05:07.058442 27252 consensus_queue.cc:1035] T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.003s
I20260925 03:05:07.051198 26937 raft_consensus.cc:2781] T 21c43af5e6844fdaacc8c625fa01fd8d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader election won for term 5
I20260925 03:05:07.134598 26937 raft_consensus.cc:686] T 21c43af5e6844fdaacc8c625fa01fd8d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 LEADER]: Becoming Leader. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:05:07.062134 26842 consensus_queue.cc:260] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:07.051326 27242 raft_consensus.cc:3037] T 43366a0a2f9d4344bf4e37ffd290ef8a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:05:07.065049 27289 raft_consensus.cc:2726] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260925 03:05:07.135664 24615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74443a0a1dc34d88a2f42e3b722db405" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
W20260925 03:05:07.048130 27126 consensus_meta.cc:287] T c88cbd39e01f44628a094af516be2833 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent flushing consensus metadata: real 0.669s user 0.010s sys 0.002s
I20260925 03:05:07.136348 26842 raft_consensus.cc:3037] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:05:07.136900 24618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d89396576884f7c8901dba54f783947" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:07.136308 27126 raft_consensus.cc:504] T c88cbd39e01f44628a094af516be2833 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:07.138134 25196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4abe0ea757e4886be4945c5001e529a" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:05:07.139452 25198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c81a1b3b616492ab6c964f45af36077" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:05:07.140013 27126 leader_election.cc:288] T c88cbd39e01f44628a094af516be2833 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 8 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:07.064040 27226 raft_consensus.cc:3037] T a7aa47c48404465b94a36fd819fe3c7a P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:05:07.056156 27255 leader_election.cc:288] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 9 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:07.068459 27265 raft_consensus.cc:2726] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260925 03:05:07.140524 27299 raft_consensus.cc:482] T b4ebd2b043024a548005c8b6b00b0449 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1bc1f2e62ab44af995165bd5116a9d43)
I20260925 03:05:07.135517 26937 consensus_queue.cc:237] T 21c43af5e6844fdaacc8c625fa01fd8d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:07.141410 27299 raft_consensus.cc:504] T b4ebd2b043024a548005c8b6b00b0449 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:07.070986 27290 raft_consensus.cc:482] T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:07.143548 27299 leader_election.cc:288] T b4ebd2b043024a548005c8b6b00b0449 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:05:07.143266 27290 raft_consensus.cc:504] T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:07.144358 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37a157996e944db2906110d87ecb0c54" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:05:07.145467 27290 leader_election.cc:288] T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:05:07.075968 27249 raft_consensus.cc:2726] T e6b6ec010caf4689b8c90449764f5cf9 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260925 03:05:07.082727 24511 leader_election.cc:302] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:05:07.066030 24588 leader_election.cc:302] T 5278a60f97da4e10aecd50a8d9a9a43c P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
W20260925 03:05:07.146520 24511 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 63847.1us
I20260925 03:05:07.146979 27301 raft_consensus.cc:2781] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260925 03:05:07.147320 27232 raft_consensus.cc:2781] T 5278a60f97da4e10aecd50a8d9a9a43c P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260925 03:05:07.147461 27301 raft_consensus.cc:482] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:05:07.147759 27232 raft_consensus.cc:482] T 5278a60f97da4e10aecd50a8d9a9a43c P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:05:07.148160 27232 raft_consensus.cc:3037] T 5278a60f97da4e10aecd50a8d9a9a43c P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:05:07.080446 24543 raft_consensus.cc:2454] T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 6.
I20260925 03:05:07.150287 27253 consensus_queue.cc:237] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:07.154469 27022 consensus_queue.cc:260] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:07.155684 27022 raft_consensus.cc:3037] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:05:07.107923 27279 raft_consensus.cc:2781] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260925 03:05:07.157344 27279 raft_consensus.cc:482] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:07.157904 27279 raft_consensus.cc:3037] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Advancing to term 8
W20260925 03:05:07.146932 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 80993.9us
I20260925 03:05:07.106966 26577 raft_consensus.cc:729] T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c [term 6 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:05:07.132558 27146 raft_consensus.cc:3037] T 25102ba08e4e49388f6176badcd10a25 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:05:07.162084 26577 consensus_queue.cc:260] T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:07.163533 26577 raft_consensus.cc:3037] T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:05:07.162943 27289 consensus_queue.cc:1035] T 550118bc1a0641ce9803bdbf43f2069f P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.002s
I20260925 03:05:07.102067 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4abe0ea757e4886be4945c5001e529a" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:07.166702 24618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b030d84c670b48d5bff337ccc0c77626" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:05:07.161016 27243 raft_consensus.cc:3032] T 74443a0a1dc34d88a2f42e3b722db405 P eeab93abfdb141a895002ec9f312296c [term 5 LEADER]: Stepping down as leader of term 5
I20260925 03:05:07.168165 27243 raft_consensus.cc:729] T 74443a0a1dc34d88a2f42e3b722db405 P eeab93abfdb141a895002ec9f312296c [term 5 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:05:07.166970 24545 raft_consensus.cc:1226] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 8. Current term is 9. Ops: []
I20260925 03:05:07.169152 27041 raft_consensus.cc:729] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [term 6 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:05:07.170105 27041 consensus_queue.cc:260] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:07.171543 24619 raft_consensus.cc:1261] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260925 03:05:07.161320 26900 consensus_queue.cc:1035] T 3d89396576884f7c8901dba54f783947 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.011s
I20260925 03:05:07.173138 27041 raft_consensus.cc:3037] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:05:07.173980 27255 raft_consensus.cc:482] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1bc1f2e62ab44af995165bd5116a9d43)
I20260925 03:05:07.168991 27243 consensus_queue.cc:260] T 74443a0a1dc34d88a2f42e3b722db405 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:07.174543 27255 raft_consensus.cc:504] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:07.175303 27301 raft_consensus.cc:3037] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Advancing to term 9
I20260925 03:05:07.119656 27104 consensus_queue.cc:260] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:05:07.103876 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 57963.3us
I20260925 03:05:07.169982 24615 raft_consensus.cc:1261] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
W20260925 03:05:07.096426 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 50376.7us
I20260925 03:05:07.183400 24514 leader_election.cc:302] T 37a157996e944db2906110d87ecb0c54 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:05:07.184028 24437 leader_election.cc:302] T a7aa47c48404465b94a36fd819fe3c7a P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:07.168522 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21c43af5e6844fdaacc8c625fa01fd8d" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:07.184792 24473 raft_consensus.cc:3037] T 21c43af5e6844fdaacc8c625fa01fd8d P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:05:07.140044 25198 raft_consensus.cc:2454] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 6.
I20260925 03:05:07.177286 24470 raft_consensus.cc:1226] T 7993f9c3e2a44f408bbf5d1e395e2306 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 5. Current term is 6. Ops: []
I20260925 03:05:07.186278 24436 leader_election.cc:302] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5; no voters:
I20260925 03:05:07.179636 24511 leader_election.cc:302] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:05:07.095594 26741 raft_consensus.cc:482] T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:07.180583 24466 raft_consensus.cc:1226] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 8. Current term is 9. Ops: []
I20260925 03:05:07.107656 24586 leader_election.cc:302] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:05:07.188930 24514 leader_election.cc:302] T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:05:07.204213 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e725b82985746b39c0cc8f671c95353" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:05:07.180317 24618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c81a1b3b616492ab6c964f45af36077" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:07.206118 24618 raft_consensus.cc:2379] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 in current term 6: Already voted for candidate eeab93abfdb141a895002ec9f312296c in this term.
I20260925 03:05:07.208328 24514 leader_election.cc:302] T 48ca6ccd98194cf2af7b5278d15bf4ea P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
W20260925 03:05:07.187374 27270 log.cc:924] Time spent T 550118bc1a0641ce9803bdbf43f2069f P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.070s user 0.001s sys 0.000s
I20260925 03:05:07.214955 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bb46bce3564841b7674e6a693bd2e6" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:05:07.215179 24514 leader_election.cc:302] T 8e5819c15eac4544a57e5ee6c9d4f117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:05:07.215304 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abd5b8dc1b7e49e3aa9af900d73ff0e7" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:05:07.175243 27243 raft_consensus.cc:3037] T 74443a0a1dc34d88a2f42e3b722db405 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:05:07.219415 24437 leader_election.cc:302] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
W20260925 03:05:07.180965 27263 log.cc:924] Time spent T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43: Append to log took a long time: real 0.063s user 0.001s sys 0.000s
I20260925 03:05:07.223487 24615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f68804eb5a244d87bb395d50683c6012" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:07.221678 24465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35992316cb8d4c028333388eff02ecde" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:07.236260 25198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6b6ec010caf4689b8c90449764f5cf9" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:05:07.201269 27299 raft_consensus.cc:2781] T 37a157996e944db2906110d87ecb0c54 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260925 03:05:07.237545 24467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be4e0aa984f4f2b81a58e382e434ad5" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:07.242262 25191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15861cd28e354598a3c6bfc4337c34bb" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:05:07.244320 24437 leader_election.cc:302] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:07.245873 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60d45b7d467d4e1ab92367f6d7d52b15" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
W20260925 03:05:07.223611 27307 log.cc:924] Time spent T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5: Append to log took a long time: real 0.086s user 0.000s sys 0.002s
I20260925 03:05:07.176645 27255 leader_election.cc:288] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:05:07.239289 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442d4a29b1d743599f4b6b4871e26668" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:05:07.241076 24467 raft_consensus.cc:2454] T 0be4e0aa984f4f2b81a58e382e434ad5 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 6.
I20260925 03:05:07.237968 24618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b627ecb4a5384ef392aa695e06599c39" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:05:07.210203 26800 raft_consensus.cc:2781] T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260925 03:05:07.211982 27126 raft_consensus.cc:2781] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Leader election won for term 9
W20260925 03:05:07.182345 26970 consensus_queue.cc:1159] T 74443a0a1dc34d88a2f42e3b722db405 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" responder_term: 5 status { last_received { term: 5 index: 4 } last_committed_idx: 4 last_received_current_leader { term: 5 index: 4 } }
I20260925 03:05:07.251042 25196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4ebd2b043024a548005c8b6b00b0449" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:05:07.251734 24614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c88cbd39e01f44628a094af516be2833" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:05:07.234527 27104 raft_consensus.cc:3037] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Advancing to term 9
I20260925 03:05:07.257221 24473 raft_consensus.cc:2454] T 21c43af5e6844fdaacc8c625fa01fd8d P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 6.
I20260925 03:05:07.223070 27273 raft_consensus.cc:2781] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260925 03:05:07.253374 24514 leader_election.cc:302] T b030d84c670b48d5bff337ccc0c77626 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
W20260925 03:05:07.305744 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 52444.1us
I20260925 03:05:07.306599 24514 leader_election.cc:302] T 100c113a9fd84cc8adf2ec7b33ea0093 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
W20260925 03:05:07.308046 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 200539us
I20260925 03:05:07.308902 27053 raft_consensus.cc:2781] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260925 03:05:07.309324 27053 raft_consensus.cc:482] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:05:07.309651 27053 raft_consensus.cc:3037] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Advancing to term 8
I20260925 03:05:07.309579 27289 consensus_queue.cc:1046] T 11bb46bce3564841b7674e6a693bd2e6 P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260925 03:05:07.310778 26970 raft_consensus.cc:3032] T 11bb46bce3564841b7674e6a693bd2e6 P eeab93abfdb141a895002ec9f312296c [term 3 LEADER]: Stepping down as leader of term 3
I20260925 03:05:07.311156 26970 raft_consensus.cc:729] T 11bb46bce3564841b7674e6a693bd2e6 P eeab93abfdb141a895002ec9f312296c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:05:07.312666 23300 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260925 03:05:07.200506 26741 raft_consensus.cc:504] T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:07.308368 24514 leader_election.cc:302] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
W20260925 03:05:07.316239 24436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 70090.6us
I20260925 03:05:07.302976 24614 raft_consensus.cc:3037] T c88cbd39e01f44628a094af516be2833 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Advancing to term 8
W20260925 03:05:07.316395 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 74469.5us
I20260925 03:05:07.317339 24437 leader_election.cc:302] T c85f5d1d3f004cf5817bb594fd8e6469 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:07.237352 27304 raft_consensus.cc:2726] T a7aa47c48404465b94a36fd819fe3c7a P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260925 03:05:07.245311 27299 raft_consensus.cc:482] T 37a157996e944db2906110d87ecb0c54 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:07.338873 24437 leader_election.cc:302] T 4e5332a5d329493991dff994e70ea92c P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:07.345008 25191 raft_consensus.cc:1261] T 7e49248643df4131a1c42e74244e05c2 P 1bc1f2e62ab44af995165bd5116a9d43 [term 8 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260925 03:05:07.345454 27272 raft_consensus.cc:2726] T c85f5d1d3f004cf5817bb594fd8e6469 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260925 03:05:07.348631 24469 raft_consensus.cc:1226] T 7b8e0d260a884ecdb20f79de46c36b19 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 6. Current term is 7. Ops: []
I20260925 03:05:07.358085 24614 raft_consensus.cc:2454] T c88cbd39e01f44628a094af516be2833 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 8.
I20260925 03:05:07.352224 27297 raft_consensus.cc:3037] T 48ca6ccd98194cf2af7b5278d15bf4ea P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:05:07.302075 27242 raft_consensus.cc:2726] T 43366a0a2f9d4344bf4e37ffd290ef8a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260925 03:05:07.359794 24437 leader_election.cc:302] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:07.312150 26970 consensus_queue.cc:260] T 11bb46bce3564841b7674e6a693bd2e6 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:07.261389 26800 raft_consensus.cc:482] T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
W20260925 03:05:07.332409 27308 log.cc:924] Time spent T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5: Append to log took a long time: real 0.079s user 0.000s sys 0.000s
I20260925 03:05:07.313293 27268 raft_consensus.cc:2726] T b030d84c670b48d5bff337ccc0c77626 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260925 03:05:07.379860 24467 raft_consensus.cc:1226] T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 5. Current term is 6. Ops: []
I20260925 03:05:07.313895 27273 raft_consensus.cc:482] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 1bc1f2e62ab44af995165bd5116a9d43)
I20260925 03:05:07.313416 27300 raft_consensus.cc:2726] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260925 03:05:07.341117 27255 raft_consensus.cc:2726] T 100c113a9fd84cc8adf2ec7b33ea0093 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260925 03:05:07.359792 27299 raft_consensus.cc:3037] T 37a157996e944db2906110d87ecb0c54 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Advancing to term 8
W20260925 03:05:07.403739 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 44040.7us
I20260925 03:05:07.404735 24437 leader_election.cc:302] T 43366a0a2f9d4344bf4e37ffd290ef8a P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:07.375744 27287 raft_consensus.cc:2726] T 4e5332a5d329493991dff994e70ea92c P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260925 03:05:07.407917 24437 leader_election.cc:302] T edcf6e6a422d4d198d4fbb0f7d3d845d P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:07.340471 27218 consensus_queue.cc:1046] T 442d4a29b1d743599f4b6b4871e26668 P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.058s
I20260925 03:05:07.354804 27265 consensus_queue.cc:1035] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.044s
I20260925 03:05:07.386016 26800 raft_consensus.cc:3037] T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:05:07.387275 27269 raft_consensus.cc:2781] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260925 03:05:07.433774 24437 leader_election.cc:302] T 9e8af2641e124f9589c5ccfdac785f63 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:07.396379 27242 raft_consensus.cc:482] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 1bc1f2e62ab44af995165bd5116a9d43)
I20260925 03:05:07.394629 27273 raft_consensus.cc:3037] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:05:07.340159 27249 consensus_queue.cc:1035] T ceab6b0fccd54e909a64f66fe81a9a1f P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.058s
I20260925 03:05:07.419382 27272 raft_consensus.cc:2726] T 43366a0a2f9d4344bf4e37ffd290ef8a P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260925 03:05:07.395449 27230 consensus_queue.cc:1046] T 7993f9c3e2a44f408bbf5d1e395e2306 P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.049s
I20260925 03:05:07.417584 26970 raft_consensus.cc:3037] T 11bb46bce3564841b7674e6a693bd2e6 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:05:07.418334 27269 raft_consensus.cc:482] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:05:07.444558 27304 raft_consensus.cc:2726] T edcf6e6a422d4d198d4fbb0f7d3d845d P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260925 03:05:07.439344 26811 raft_consensus.cc:2781] T 8e5819c15eac4544a57e5ee6c9d4f117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260925 03:05:07.465428 24437 leader_election.cc:302] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:07.439064 27231 consensus_queue.cc:1035] T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.023s
I20260925 03:05:07.459339 27272 raft_consensus.cc:2726] T 9e8af2641e124f9589c5ccfdac785f63 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20260925 03:05:07.478986 27305 log.cc:924] Time spent T 71f461a6e2ff434993b78a2d077f0abd P 1bc1f2e62ab44af995165bd5116a9d43: Append to log took a long time: real 0.147s user 0.000s sys 0.001s
I20260925 03:05:07.469116 27254 raft_consensus.cc:504] T e6b6ec010caf4689b8c90449764f5cf9 P 1bc1f2e62ab44af995165bd5116a9d43 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:07.533119 25198 raft_consensus.cc:2379] T e6b6ec010caf4689b8c90449764f5cf9 P 1bc1f2e62ab44af995165bd5116a9d43 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 7: Already voted for candidate 1bc1f2e62ab44af995165bd5116a9d43 in this term.
I20260925 03:05:07.471895 27304 raft_consensus.cc:3037] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:05:07.538460 27190 raft_consensus.cc:504] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:07.540580 27190 leader_election.cc:288] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 8 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:07.541363 24544 raft_consensus.cc:1226] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 7. Current term is 8. Ops: []
I20260925 03:05:07.541499 27254 leader_election.cc:288] T e6b6ec010caf4689b8c90449764f5cf9 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 7 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:05:07.486951 27232 raft_consensus.cc:504] T 5278a60f97da4e10aecd50a8d9a9a43c P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:07.538429 27301 raft_consensus.cc:504] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:07.556064 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6b6ec010caf4689b8c90449764f5cf9" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:07.556897 24472 raft_consensus.cc:3037] T e6b6ec010caf4689b8c90449764f5cf9 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:05:07.563436 27297 raft_consensus.cc:2726] T 48ca6ccd98194cf2af7b5278d15bf4ea P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260925 03:05:07.566308 24617 raft_consensus.cc:2454] T 37a157996e944db2906110d87ecb0c54 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 8.
I20260925 03:05:07.502506 26741 leader_election.cc:288] T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:05:07.509325 24614 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "48ca6ccd98194cf2af7b5278d15bf4ea"
dest_uuid: "eeab93abfdb141a895002ec9f312296c"
mode: GRACEFUL
new_leader_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
from {username='slave'} at 127.0.0.1:44570
I20260925 03:05:07.570421 24614 raft_consensus.cc:595] T 48ca6ccd98194cf2af7b5278d15bf4ea P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Received request to transfer leadership to TS a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:05:07.571018 24614 raft_consensus.cc:603] T 48ca6ccd98194cf2af7b5278d15bf4ea P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260925 03:05:07.450147 27242 raft_consensus.cc:504] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:07.543749 27232 leader_election.cc:288] T 5278a60f97da4e10aecd50a8d9a9a43c P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 6 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:05:07.464334 27269 raft_consensus.cc:3037] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Advancing to term 8
I20260925 03:05:07.573213 24621 raft_consensus.cc:2454] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 7.
I20260925 03:05:07.570611 27299 raft_consensus.cc:504] T 37a157996e944db2906110d87ecb0c54 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:07.494818 27279 raft_consensus.cc:504] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:07.523052 24618 raft_consensus.cc:2454] T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 7.
I20260925 03:05:07.578514 25196 raft_consensus.cc:1261] T 04e068a0f80941dc9138cee063f53bab P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260925 03:05:07.573348 27242 leader_election.cc:288] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:05:07.586033 24619 raft_consensus.cc:2454] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 7.
W20260925 03:05:07.583796 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 275476us
I20260925 03:05:07.616161 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7993f9c3e2a44f408bbf5d1e395e2306" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:05:07.605207 24514 leader_election.cc:302] T 0be4e0aa984f4f2b81a58e382e434ad5 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:07.617381 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b8e0d260a884ecdb20f79de46c36b19" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:05:07.629360 24542 raft_consensus.cc:1226] T c88cbd39e01f44628a094af516be2833 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 7. Current term is 8. Ops: []
I20260925 03:05:07.630414 24514 leader_election.cc:302] T 7993f9c3e2a44f408bbf5d1e395e2306 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:07.603372 24472 raft_consensus.cc:2454] T e6b6ec010caf4689b8c90449764f5cf9 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 7.
I20260925 03:05:07.641556 25165 leader_election.cc:302] T e6b6ec010caf4689b8c90449764f5cf9 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5; no voters:
I20260925 03:05:07.649667 25199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71f461a6e2ff434993b78a2d077f0abd" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:05:07.650265 24436 leader_election.cc:302] T 37a157996e944db2906110d87ecb0c54 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:05:07.651855 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:07.655296 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9c33fd100ff44c1a3247319a324a044" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:05:07.655829 24472 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "550118bc1a0641ce9803bdbf43f2069f"
dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
mode: GRACEFUL
new_leader_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
from {username='slave'} at 127.0.0.1:38832
I20260925 03:05:07.652905 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37a157996e944db2906110d87ecb0c54" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:05:07.658087 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d89396576884f7c8901dba54f783947" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:07.651960 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74443a0a1dc34d88a2f42e3b722db405" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:05:07.658469 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b030d84c670b48d5bff337ccc0c77626" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:05:07.655973 24544 raft_consensus.cc:2362] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 7. Re-sending same reply.
I20260925 03:05:07.591950 27246 raft_consensus.cc:482] T 29637f5a4a504ce6ba29033ccd0cd716 P 1bc1f2e62ab44af995165bd5116a9d43 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:07.596412 26946 raft_consensus.cc:2726] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20260925 03:05:07.716673 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 86324.4us
I20260925 03:05:07.724746 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e068a0f80941dc9138cee063f53bab" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:07.725611 24541 raft_consensus.cc:2354] T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1bc1f2e62ab44af995165bd5116a9d43 for earlier term 5. Current term is 6.
I20260925 03:05:07.725519 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6b6ec010caf4689b8c90449764f5cf9" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:07.726236 24540 raft_consensus.cc:2454] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 6.
W20260925 03:05:07.726577 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 394739us
I20260925 03:05:07.727088 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6b6ec010caf4689b8c90449764f5cf9" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:05:07.726614 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367277daf8d04778b3fa4ed94d71580d" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:07.649477 27190 raft_consensus.cc:3037] T 0be4e0aa984f4f2b81a58e382e434ad5 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:05:07.575589 27053 raft_consensus.cc:504] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:07.727703 24586 leader_election.cc:302] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:05:07.714315 26995 raft_consensus.cc:2781] T e6b6ec010caf4689b8c90449764f5cf9 P 1bc1f2e62ab44af995165bd5116a9d43 [term 7 FOLLOWER]: Leader election won for term 7
I20260925 03:05:07.730571 24437 leader_election.cc:302] T 7b8e0d260a884ecdb20f79de46c36b19 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:07.734084 24539 raft_consensus.cc:1261] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260925 03:05:07.731293 24467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5278a60f97da4e10aecd50a8d9a9a43c" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:07.727671 24543 raft_consensus.cc:3037] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:05:07.743111 24545 raft_consensus.cc:1261] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260925 03:05:07.639923 27022 consensus_queue.cc:1035] T 04e068a0f80941dc9138cee063f53bab P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.086s
W20260925 03:05:07.747917 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 494580us
I20260925 03:05:07.728566 27255 raft_consensus.cc:2726] T 7993f9c3e2a44f408bbf5d1e395e2306 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260925 03:05:07.777817 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 50197.8us
I20260925 03:05:07.715062 27267 raft_consensus.cc:2781] T 37a157996e944db2906110d87ecb0c54 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Leader election won for term 8
I20260925 03:05:07.853106 24620 raft_consensus.cc:1226] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 7. Current term is 8. Ops: [7.4-7.4]
I20260925 03:05:07.833154 24467 raft_consensus.cc:2379] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 5: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:05:07.742828 27304 raft_consensus.cc:2726] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260925 03:05:07.835608 26970 raft_consensus.cc:2781] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260925 03:05:07.574864 27273 raft_consensus.cc:504] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:07.744148 27301 leader_election.cc:288] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 9 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
W20260925 03:05:07.847724 25163 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 125291us
I20260925 03:05:07.808751 27126 raft_consensus.cc:686] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 LEADER]: Becoming Leader. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:05:07.857174 24621 raft_consensus.cc:1226] T c88cbd39e01f44628a094af516be2833 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 7. Current term is 8. Ops: []
I20260925 03:05:07.861634 24543 raft_consensus.cc:2454] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 7.
I20260925 03:05:07.875911 25163 leader_election.cc:398] T 04e068a0f80941dc9138cee063f53bab P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 5 pre-election: Vote denied by peer a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389) with higher term. Message: Invalid argument: T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1bc1f2e62ab44af995165bd5116a9d43 for earlier term 5. Current term is 6.
I20260925 03:05:07.875993 25196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43"
I20260925 03:05:07.875118 24467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:07.878532 25163 leader_election.cc:401] T 04e068a0f80941dc9138cee063f53bab P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20260925 03:05:07.879997 24588 leader_election.cc:302] T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:05:07.883889 26970 raft_consensus.cc:482] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Starting leader election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:05:07.884393 26970 raft_consensus.cc:3037] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Advancing to term 8
W20260925 03:05:07.891410 24514 leader_election.cc:334] T 5278a60f97da4e10aecd50a8d9a9a43c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187): Timed out: RequestConsensusVote RPC to 127.22.193.1:42187 timed out after 1.682s (SENT)
I20260925 03:05:07.891814 24514 leader_election.cc:302] T 5278a60f97da4e10aecd50a8d9a9a43c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:07.872603 26811 raft_consensus.cc:482] T 8e5819c15eac4544a57e5ee6c9d4f117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:07.729374 27246 raft_consensus.cc:504] T 29637f5a4a504ce6ba29033ccd0cd716 P 1bc1f2e62ab44af995165bd5116a9d43 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:07.897455 25199 raft_consensus.cc:1226] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 5. Current term is 6. Ops: []
I20260925 03:05:07.899302 27246 leader_election.cc:288] T 29637f5a4a504ce6ba29033ccd0cd716 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:07.900787 25196 raft_consensus.cc:3037] T 29637f5a4a504ce6ba29033ccd0cd716 P 1bc1f2e62ab44af995165bd5116a9d43 [term 8 FOLLOWER]: Advancing to term 9
I20260925 03:05:07.897395 26811 raft_consensus.cc:3037] T 8e5819c15eac4544a57e5ee6c9d4f117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Advancing to term 8
I20260925 03:05:07.904115 27218 raft_consensus.cc:3032] T 7993f9c3e2a44f408bbf5d1e395e2306 P eeab93abfdb141a895002ec9f312296c [term 5 LEADER]: Stepping down as leader of term 5
I20260925 03:05:07.904366 27268 raft_consensus.cc:2726] T 5278a60f97da4e10aecd50a8d9a9a43c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260925 03:05:07.906358 24514 leader_election.cc:334] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187): Timed out: RequestConsensusVote RPC to 127.22.193.1:42187 timed out after 1.777s (SENT)
W20260925 03:05:07.917718 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 187262us
I20260925 03:05:07.918633 24437 leader_election.cc:302] T 7993f9c3e2a44f408bbf5d1e395e2306 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:07.917744 24472 raft_consensus.cc:595] T 550118bc1a0641ce9803bdbf43f2069f P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Received request to transfer leadership to TS a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:05:07.919670 24472 raft_consensus.cc:603] T 550118bc1a0641ce9803bdbf43f2069f P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260925 03:05:07.920735 25163 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 44968.1us
I20260925 03:05:07.922277 25163 leader_election.cc:302] T 367277daf8d04778b3fa4ed94d71580d P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:07.926893 27273 leader_election.cc:288] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 election: Requested vote from peers eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:05:07.929733 27279 leader_election.cc:288] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 8 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:07.931095 26800 raft_consensus.cc:504] T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:05:07.934343 24436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 53369.3us
W20260925 03:05:07.934689 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 79785.4us
W20260925 03:05:07.934795 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 54882.6us
I20260925 03:05:07.936152 24588 leader_election.cc:302] T 21c43af5e6844fdaacc8c625fa01fd8d P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:05:07.921901 27267 raft_consensus.cc:686] T 37a157996e944db2906110d87ecb0c54 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 LEADER]: Becoming Leader. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:05:07.933082 27249 consensus_queue.cc:1046] T c88cbd39e01f44628a094af516be2833 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.005s
I20260925 03:05:07.939924 24437 leader_election.cc:302] T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:07.941287 24437 leader_election.cc:302] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:07.942621 27267 consensus_queue.cc:237] T 37a157996e944db2906110d87ecb0c54 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:07.945071 25198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c81a1b3b616492ab6c964f45af36077" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43"
I20260925 03:05:07.951459 27272 raft_consensus.cc:2726] T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260925 03:05:07.952268 27126 consensus_queue.cc:237] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:05:07.954555 27313 log.cc:924] Time spent T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.184s user 0.002s sys 0.008s
I20260925 03:05:07.933347 27296 raft_consensus.cc:2726] T 7b8e0d260a884ecdb20f79de46c36b19 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260925 03:05:07.922946 27185 raft_consensus.cc:2726] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260925 03:05:07.961347 27053 leader_election.cc:288] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 8 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:05:07.967818 25196 raft_consensus.cc:2454] T 29637f5a4a504ce6ba29033ccd0cd716 P 1bc1f2e62ab44af995165bd5116a9d43 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 9.
I20260925 03:05:07.971971 25198 raft_consensus.cc:2454] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 6.
I20260925 03:05:07.972939 27052 raft_consensus.cc:2781] T 21c43af5e6844fdaacc8c625fa01fd8d P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Leader election won for term 6
I20260925 03:05:07.961975 27077 raft_consensus.cc:2726] T 7993f9c3e2a44f408bbf5d1e395e2306 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260925 03:05:07.977263 27299 leader_election.cc:288] T 37a157996e944db2906110d87ecb0c54 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 8 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:07.955049 27304 consensus_queue.cc:1046] T 293ef69caf514ac98510a68495f54eda P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.023s
I20260925 03:05:07.963824 27279 raft_consensus.cc:482] T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:07.904508 27218 raft_consensus.cc:729] T 7993f9c3e2a44f408bbf5d1e395e2306 P eeab93abfdb141a895002ec9f312296c [term 5 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:05:07.996476 27268 raft_consensus.cc:482] T 1414dcfe53f64a78ae413531eb8ec117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:07.920354 27246 raft_consensus.cc:482] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:08.018285 27052 raft_consensus.cc:686] T 21c43af5e6844fdaacc8c625fa01fd8d P eeab93abfdb141a895002ec9f312296c [term 6 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:05:07.979859 27252 raft_consensus.cc:3032] T 442d4a29b1d743599f4b6b4871e26668 P eeab93abfdb141a895002ec9f312296c [term 7 LEADER]: Stepping down as leader of term 7
I20260925 03:05:08.018604 27268 raft_consensus.cc:504] T 1414dcfe53f64a78ae413531eb8ec117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:07.952237 27272 raft_consensus.cc:2726] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260925 03:05:07.965005 24467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:07.913056 27259 consensus_queue.cc:1046] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 8.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.072s
I20260925 03:05:08.019454 27052 consensus_queue.cc:237] T 21c43af5e6844fdaacc8c625fa01fd8d P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:07.977705 26800 leader_election.cc:288] T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 6 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:08.020726 27259 raft_consensus.cc:3032] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [term 8 LEADER]: Stepping down as leader of term 8
I20260925 03:05:08.021103 27259 raft_consensus.cc:729] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [term 8 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:05:08.021884 27259 consensus_queue.cc:260] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:05:07.987533 27320 log.cc:924] Time spent T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5: Append to log took a long time: real 0.086s user 0.000s sys 0.001s
I20260925 03:05:08.023124 27328 raft_consensus.cc:482] T 43366a0a2f9d4344bf4e37ffd290ef8a P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:08.023404 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5278a60f97da4e10aecd50a8d9a9a43c" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
W20260925 03:05:07.987900 24585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.4:43979, user_credentials={real_user=slave}} blocked reactor thread for 94903.5us
I20260925 03:05:07.988759 26762 raft_consensus.cc:2781] T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260925 03:05:08.023741 27328 raft_consensus.cc:504] T 43366a0a2f9d4344bf4e37ffd290ef8a P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:08.024382 26762 raft_consensus.cc:482] T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:05:08.024789 26762 raft_consensus.cc:3037] T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:05:08.024878 27304 raft_consensus.cc:3032] T 293ef69caf514ac98510a68495f54eda P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 LEADER]: Stepping down as leader of term 5
I20260925 03:05:08.025247 27304 raft_consensus.cc:729] T 293ef69caf514ac98510a68495f54eda P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:05:08.025991 27304 consensus_queue.cc:260] T 293ef69caf514ac98510a68495f54eda P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 5.8, Last appended by leader: 8, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:08.027290 27304 raft_consensus.cc:3037] T 293ef69caf514ac98510a68495f54eda P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:05:07.998713 26946 consensus_queue.cc:1046] T c88cbd39e01f44628a094af516be2833 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.030s
I20260925 03:05:08.015901 27282 raft_consensus.cc:3032] T c88cbd39e01f44628a094af516be2833 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 LEADER]: Stepping down as leader of term 7
I20260925 03:05:08.029546 27282 raft_consensus.cc:729] T c88cbd39e01f44628a094af516be2833 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:05:07.989382 26970 raft_consensus.cc:504] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:08.030539 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:05:08.031122 24620 raft_consensus.cc:3037] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Advancing to term 9
I20260925 03:05:07.992575 27190 raft_consensus.cc:2726] T 0be4e0aa984f4f2b81a58e382e434ad5 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260925 03:05:08.015888 27255 raft_consensus.cc:482] T 3d89396576884f7c8901dba54f783947 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:07.991281 26803 raft_consensus.cc:2726] T 04e068a0f80941dc9138cee063f53bab P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389) with higher term. Message: Invalid argument: T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1bc1f2e62ab44af995165bd5116a9d43 for earlier term 5. Current term is 6.
I20260925 03:05:08.032217 27255 raft_consensus.cc:504] T 3d89396576884f7c8901dba54f783947 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:08.032485 24368 catalog_manager.cc:5495] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 reported cstate change: term changed from 3 to 5, leader changed from eeab93abfdb141a895002ec9f312296c (127.22.193.3) to 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4). New cstate: current_term: 5 leader_uuid: "1bc1f2e62ab44af995165bd5116a9d43" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:05:08.033528 24614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f1fac028a144058b92331c0d9b44ebf" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:05:08.034303 27255 leader_election.cc:288] T 3d89396576884f7c8901dba54f783947 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:08.034024 24368 catalog_manager.cc:5495] T 293ef69caf514ac98510a68495f54eda P 869b0a8575b54d08ae2817fad5dd42d5 reported cstate change: term changed from 4 to 5, leader changed from 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) to 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1). New cstate: current_term: 5 leader_uuid: "869b0a8575b54d08ae2817fad5dd42d5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260925 03:05:08.034758 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74443a0a1dc34d88a2f42e3b722db405" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:05:08.035702 24471 raft_consensus.cc:1261] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260925 03:05:08.017722 27218 consensus_queue.cc:260] T 7993f9c3e2a44f408bbf5d1e395e2306 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:08.025910 27328 leader_election.cc:288] T 43366a0a2f9d4344bf4e37ffd290ef8a P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:05:08.018432 27339 raft_consensus.cc:2726] T 367277daf8d04778b3fa4ed94d71580d P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260925 03:05:08.038681 27218 raft_consensus.cc:3037] T 7993f9c3e2a44f408bbf5d1e395e2306 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:05:08.035948 24368 catalog_manager.cc:5495] T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5 reported cstate change: term changed from 4 to 5, leader changed from 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1) to eeab93abfdb141a895002ec9f312296c (127.22.193.3). New cstate: current_term: 5 leader_uuid: "eeab93abfdb141a895002ec9f312296c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } }
I20260925 03:05:08.043053 27246 raft_consensus.cc:504] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:08.045050 27246 leader_election.cc:288] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:05:08.045609 24467 raft_consensus.cc:2454] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 8.
I20260925 03:05:08.047189 26946 consensus_queue.cc:1046] T ceab6b0fccd54e909a64f66fe81a9a1f P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260925 03:05:08.048237 27272 raft_consensus.cc:3032] T ceab6b0fccd54e909a64f66fe81a9a1f P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 LEADER]: Stepping down as leader of term 7
I20260925 03:05:08.048591 27272 raft_consensus.cc:729] T ceab6b0fccd54e909a64f66fe81a9a1f P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:05:08.049449 27272 consensus_queue.cc:260] T ceab6b0fccd54e909a64f66fe81a9a1f P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:08.050294 24613 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9e8af2641e124f9589c5ccfdac785f63"
dest_uuid: "eeab93abfdb141a895002ec9f312296c"
mode: GRACEFUL
new_leader_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
from {username='slave'} at 127.0.0.1:44570
I20260925 03:05:08.050661 27272 raft_consensus.cc:3037] T ceab6b0fccd54e909a64f66fe81a9a1f P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Advancing to term 8
I20260925 03:05:08.050874 24613 raft_consensus.cc:595] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Received request to transfer leadership to TS a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:05:08.051216 24613 raft_consensus.cc:603] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260925 03:05:08.029491 24618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71f461a6e2ff434993b78a2d077f0abd" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:08.051966 27322 raft_consensus.cc:482] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:08.019398 27252 raft_consensus.cc:729] T 442d4a29b1d743599f4b6b4871e26668 P eeab93abfdb141a895002ec9f312296c [term 7 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:05:08.052630 27322 raft_consensus.cc:504] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:08.053160 27252 consensus_queue.cc:260] T 442d4a29b1d743599f4b6b4871e26668 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:08.054463 27252 raft_consensus.cc:3037] T 442d4a29b1d743599f4b6b4871e26668 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Advancing to term 8
I20260925 03:05:08.054991 27322 leader_election.cc:288] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:07.981797 26811 raft_consensus.cc:504] T 8e5819c15eac4544a57e5ee6c9d4f117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:08.031970 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5332a5d329493991dff994e70ea92c" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:05:08.058421 26811 leader_election.cc:288] T 8e5819c15eac4544a57e5ee6c9d4f117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 8 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:08.029460 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9c33fd100ff44c1a3247319a324a044" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:08.059152 24470 raft_consensus.cc:2454] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 7.
I20260925 03:05:08.022946 24465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceab6b0fccd54e909a64f66fe81a9a1f" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
W20260925 03:05:07.990366 25165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 47334.4us
I20260925 03:05:08.060101 24614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c81a1b3b616492ab6c964f45af36077" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:05:08.033089 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37a157996e944db2906110d87ecb0c54" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:05:08.060920 24614 raft_consensus.cc:2379] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 in current term 6: Already voted for candidate eeab93abfdb141a895002ec9f312296c in this term.
I20260925 03:05:08.061061 24617 raft_consensus.cc:2379] T 37a157996e944db2906110d87ecb0c54 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 8: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:05:08.031379 26970 leader_election.cc:288] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 8 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:05:08.024009 24472 raft_consensus.cc:3037] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:05:08.014585 24511 leader_election.cc:302] T c88cbd39e01f44628a094af516be2833 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
W20260925 03:05:08.061992 24511 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 47503.8us
I20260925 03:05:08.062614 24511 leader_election.cc:302] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:05:08.016938 27279 raft_consensus.cc:504] T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:08.065433 27279 leader_election.cc:288] T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:05:08.066062 27279 raft_consensus.cc:2781] T c88cbd39e01f44628a094af516be2833 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Leader election won for term 8
I20260925 03:05:08.070695 24619 raft_consensus.cc:2454] T 74443a0a1dc34d88a2f42e3b722db405 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 6.
I20260925 03:05:08.023097 27259 raft_consensus.cc:3037] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Advancing to term 9
I20260925 03:05:08.074361 27269 raft_consensus.cc:504] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:05:07.993826 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 58543.1us
I20260925 03:05:08.063436 24511 leader_election.cc:302] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
W20260925 03:05:08.080399 25162 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 204901us
I20260925 03:05:08.062885 24436 leader_election.cc:302] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5; no voters:
I20260925 03:05:08.081426 24619 raft_consensus.cc:1226] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 7. Current term is 8. Ops: []
I20260925 03:05:08.084321 24472 raft_consensus.cc:2454] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 6.
I20260925 03:05:08.085829 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35992316cb8d4c028333388eff02ecde" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:08.063338 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "293ef69caf514ac98510a68495f54eda" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:08.086894 24473 raft_consensus.cc:2454] T 293ef69caf514ac98510a68495f54eda P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 6.
I20260925 03:05:08.089625 24471 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "cc2df40db08649c9a75e90000b878559"
dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
mode: GRACEFUL
new_leader_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
from {username='slave'} at 127.0.0.1:38832
I20260925 03:05:08.090660 24471 raft_consensus.cc:595] T cc2df40db08649c9a75e90000b878559 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Received request to transfer leadership to TS a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:05:08.091573 24471 raft_consensus.cc:603] T cc2df40db08649c9a75e90000b878559 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260925 03:05:08.080307 24586 leader_election.cc:302] T a4abe0ea757e4886be4945c5001e529a P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 1bc1f2e62ab44af995165bd5116a9d43, a04a37fc4aa7460cbfdbeeb4e451e1f5
W20260925 03:05:08.101289 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 160084us
I20260925 03:05:08.102106 24437 leader_election.cc:302] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:08.101919 24511 leader_election.cc:302] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:05:08.102874 25165 leader_election.cc:302] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5; no voters:
I20260925 03:05:08.076542 27269 leader_election.cc:288] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 8 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:08.125187 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43366a0a2f9d4344bf4e37ffd290ef8a" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:08.133227 24470 raft_consensus.cc:2454] T 43366a0a2f9d4344bf4e37ffd290ef8a P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 4.
I20260925 03:05:08.126397 24511 leader_election.cc:302] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 1bc1f2e62ab44af995165bd5116a9d43, eeab93abfdb141a895002ec9f312296c
W20260925 03:05:08.129504 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 49277.9us
I20260925 03:05:08.131000 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:08.101362 27249 raft_consensus.cc:2781] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader election won for term 6
I20260925 03:05:08.088868 27279 raft_consensus.cc:686] T c88cbd39e01f44628a094af516be2833 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 LEADER]: Becoming Leader. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:05:08.100557 27301 raft_consensus.cc:2781] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Leader election won for term 7
I20260925 03:05:08.147850 26900 raft_consensus.cc:482] T 48ca6ccd98194cf2af7b5278d15bf4ea P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:08.158639 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be4e0aa984f4f2b81a58e382e434ad5" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:08.151444 27041 consensus_queue.cc:1035] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260925 03:05:08.130167 27339 raft_consensus.cc:2781] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260925 03:05:08.153496 24368 catalog_manager.cc:5495] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 reported cstate change: term changed from 3 to 6, leader changed from a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2) to eeab93abfdb141a895002ec9f312296c (127.22.193.3). New cstate: current_term: 6 leader_uuid: "eeab93abfdb141a895002ec9f312296c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } }
I20260925 03:05:08.145563 24511 leader_election.cc:302] T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
W20260925 03:05:08.202662 24369 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39382 (request call id 190) took 26940 ms (client timeout 15000 ms). Trace:
W20260925 03:05:08.155520 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 53488.4us
I20260925 03:05:08.126066 27255 raft_consensus.cc:2781] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Leader election won for term 7
I20260925 03:05:08.126096 27077 raft_consensus.cc:2726] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260925 03:05:08.100791 27268 leader_election.cc:288] T 1414dcfe53f64a78ae413531eb8ec117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:08.150122 27291 consensus_queue.cc:1046] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260925 03:05:08.159312 24547 raft_consensus.cc:2454] T 0be4e0aa984f4f2b81a58e382e434ad5 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 6.
I20260925 03:05:08.216120 24546 raft_consensus.cc:1226] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 LEADER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20260925 03:05:08.159561 24588 leader_election.cc:302] T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:05:08.133314 26970 raft_consensus.cc:2726] T a4abe0ea757e4886be4945c5001e529a P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260925 03:05:08.133826 27190 raft_consensus.cc:2781] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Leader election won for term 7
I20260925 03:05:08.220930 24469 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1358) took 1012 ms. Trace:
W20260925 03:05:08.161031 25163 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 238847us
I20260925 03:05:08.133689 27273 consensus_queue.cc:1046] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.031s
I20260925 03:05:08.143790 26811 raft_consensus.cc:2726] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260925 03:05:08.185458 27279 consensus_queue.cc:237] T c88cbd39e01f44628a094af516be2833 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:08.162487 27009 consensus_queue.cc:1046] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: INVALID_TERM, Last received: 8.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.024s
I20260925 03:05:08.166956 26900 raft_consensus.cc:504] T 48ca6ccd98194cf2af7b5278d15bf4ea P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:08.239923 26900 leader_election.cc:288] T 48ca6ccd98194cf2af7b5278d15bf4ea P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:05:08.193893 24541 raft_consensus.cc:1226] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 7. Current term is 8. Ops: []
I20260925 03:05:08.242130 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "293ef69caf514ac98510a68495f54eda" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
W20260925 03:05:08.242786 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 83307.7us
I20260925 03:05:08.243252 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43366a0a2f9d4344bf4e37ffd290ef8a" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:08.243844 24541 raft_consensus.cc:2454] T 43366a0a2f9d4344bf4e37ffd290ef8a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 4.
I20260925 03:05:08.195299 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5278a60f97da4e10aecd50a8d9a9a43c" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:05:08.244933 24539 raft_consensus.cc:3037] T 5278a60f97da4e10aecd50a8d9a9a43c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Advancing to term 6
I20260925 03:05:08.245162 24588 leader_election.cc:302] T 5278a60f97da4e10aecd50a8d9a9a43c P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:05:08.245848 26970 raft_consensus.cc:2781] T 5278a60f97da4e10aecd50a8d9a9a43c P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Leader election won for term 6
I20260925 03:05:08.246331 26970 raft_consensus.cc:686] T 5278a60f97da4e10aecd50a8d9a9a43c P eeab93abfdb141a895002ec9f312296c [term 6 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:05:08.247117 26970 consensus_queue.cc:237] T 5278a60f97da4e10aecd50a8d9a9a43c P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:08.196367 24544 raft_consensus.cc:1261] T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260925 03:05:08.249974 24544 raft_consensus.cc:1261] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260925 03:05:08.198688 24368 catalog_manager.cc:5495] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 reported cstate change: term changed from 1 to 8, leader changed from 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1) to eeab93abfdb141a895002ec9f312296c (127.22.193.3). New cstate: current_term: 8 leader_uuid: "eeab93abfdb141a895002ec9f312296c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } }
I20260925 03:05:08.106674 27282 consensus_queue.cc:260] T c88cbd39e01f44628a094af516be2833 P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:08.247233 24620 raft_consensus.cc:2454] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 9.
I20260925 03:05:08.164529 27339 raft_consensus.cc:482] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:05:08.262909 24588 leader_election.cc:302] T 43366a0a2f9d4344bf4e37ffd290ef8a P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
W20260925 03:05:08.206707 24511 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 61230.2us
I20260925 03:05:08.143226 26995 raft_consensus.cc:686] T e6b6ec010caf4689b8c90449764f5cf9 P 1bc1f2e62ab44af995165bd5116a9d43 [term 7 LEADER]: Becoming Leader. State: Replica: 1bc1f2e62ab44af995165bd5116a9d43, State: Running, Role: LEADER
I20260925 03:05:08.165926 27249 raft_consensus.cc:686] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 LEADER]: Becoming Leader. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:05:08.269086 27249 consensus_queue.cc:237] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:08.268638 26995 consensus_queue.cc:237] T e6b6ec010caf4689b8c90449764f5cf9 P 1bc1f2e62ab44af995165bd5116a9d43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 5.8, Last appended by leader: 8, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:08.207156 24437 leader_election.cc:302] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:05:08.223271 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceab6b0fccd54e909a64f66fe81a9a1f" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:05:08.198587 27301 raft_consensus.cc:686] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 LEADER]: Becoming Leader. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
W20260925 03:05:08.203033 24369 rpcz_store.cc:269] 0925 03:04:41.245798 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:41.262040 (+ 16242us) service_pool.cc:227] Handling call
0925 03:05:08.175448 (+26913408us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0925 03:05:05.955956 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0925 03:05:05.956618 (+ 662us) write_op.cc:432] Acquiring schema lock in shared mode
0925 03:05:05.956675 (+ 57us) write_op.cc:435] Acquired schema lock
0925 03:05:05.956712 (+ 37us) tablet.cc:660] Decoding operations
0925 03:05:05.958009 (+ 1297us) write_op.cc:620] Acquiring the partition lock for write op
0925 03:05:05.958107 (+ 98us) write_op.cc:641] Partition lock acquired for write op
0925 03:05:05.958142 (+ 35us) tablet.cc:683] Acquiring locks for 13 operations
0925 03:05:06.056908 (+ 98766us) tablet.cc:699] Row locks acquired
0925 03:05:06.056945 (+ 37us) write_op.cc:260] PREPARE: finished
0925 03:05:06.057125 (+ 180us) write_op.cc:270] Start()
0925 03:05:06.057278 (+ 153us) write_op.cc:276] Timestamp: P: 1790305506057059 usec, L: 0
0925 03:05:06.057317 (+ 39us) op_driver.cc:331] REPLICATION: starting
0925 03:05:06.117191 (+ 59874us) log.cc:842] Serialized 5149 byte log entry
0925 03:05:06.867446 (+750255us) op_driver.cc:444] REPLICATION: finished
0925 03:05:07.448461 (+581015us) write_op.cc:301] APPLY: starting
0925 03:05:07.448582 (+ 121us) tablet.cc:1365] starting BulkCheckPresence
0925 03:05:07.448877 (+ 295us) tablet.cc:1368] finished BulkCheckPresence
0925 03:05:07.448989 (+ 112us) tablet.cc:1370] starting ApplyRowOperation cycle
0925 03:05:07.478666 (+ 29677us) tablet.cc:1381] finished ApplyRowOperation cycle
0925 03:05:07.478989 (+ 323us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=13
0925 03:05:07.479047 (+ 58us) write_op.cc:312] APPLY: finished
0925 03:05:07.479437 (+ 390us) log.cc:842] Serialized 94 byte log entry
0925 03:05:07.479618 (+ 181us) write_op.cc:489] Releasing partition, row and schema locks
0925 03:05:07.479770 (+ 152us) write_op.cc:454] Released schema lock
0925 03:05:07.479843 (+ 73us) write_op.cc:341] FINISH: Updating metrics
0925 03:05:07.480211 (+ 368us) write_op.cc:489] Releasing partition, row and schema locks
0925 03:05:07.480240 (+ 29us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":179,"reported_tablets":49,"child_traces":[["op",{"apply.queue_time_us":580685,"apply.run_cpu_time_us":4814,"apply.run_wall_time_us":32245,"num_ops":13,"prepare.queue_time_us":283170,"prepare.run_cpu_time_us":10566,"prepare.run_wall_time_us":243409,"raft.queue_time_us":311546,"raft.run_cpu_time_us":111461,"raft.run_wall_time_us":440355,"replication_time_us":810029,"thread_start_us":641449,"threads_started":4,"wal-append.queue_time_us":101571}]]}
I20260925 03:05:08.233748 27291 raft_consensus.cc:3032] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c [term 7 LEADER]: Stepping down as leader of term 7
I20260925 03:05:08.235216 27190 raft_consensus.cc:686] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 LEADER]: Becoming Leader. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:05:08.243667 27022 raft_consensus.cc:2781] T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260925 03:05:08.221319 24469 rpcz_store.cc:276] 0925 03:05:07.208817 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:05:07.387887 (+179070us) service_pool.cc:227] Handling call
0925 03:05:07.388273 (+ 386us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:05:07.397159 (+ 8886us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:05:07.397194 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:05:07.401519 (+ 4325us) log.cc:842] Serialized 26 byte log entry
0925 03:05:07.857635 (+456116us) raft_consensus.cc:1631] Marking committed up to 3
0925 03:05:07.922656 (+ 65021us) log.cc:842] Serialized 14 byte log entry
0925 03:05:07.953402 (+ 30746us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:05:07.953509 (+ 107us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:05:08.220710 (+267201us) raft_consensus.cc:1671] finished
0925 03:05:08.220733 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:05:08.220865 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":56489,"prepare.queue_time_us":64755,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":45,"raft.queue_time_us":34763,"raft.run_cpu_time_us":247,"raft.run_wall_time_us":6111,"thread_start_us":406493,"threads_started":1,"wal-append.queue_time_us":715710}
I20260925 03:05:08.244632 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:08.204949 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9c33fd100ff44c1a3247319a324a044" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:08.288772 24542 raft_consensus.cc:2454] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 7.
I20260925 03:05:08.253535 24368 catalog_manager.cc:5495] T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5 reported cstate change: term changed from 3 to 6, leader changed from eeab93abfdb141a895002ec9f312296c (127.22.193.3) to 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4). New cstate: current_term: 6 leader_uuid: "1bc1f2e62ab44af995165bd5116a9d43" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:05:08.290149 24368 catalog_manager.cc:5495] T c5aec38e66a947c681e6c2a72b05e6ca P 869b0a8575b54d08ae2817fad5dd42d5 reported cstate change: term changed from 3 to 9. New cstate: current_term: 9 leader_uuid: "eeab93abfdb141a895002ec9f312296c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:05:08.215684 27269 raft_consensus.cc:2781] T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader election won for term 6
I20260925 03:05:08.263496 24539 raft_consensus.cc:2454] T 5278a60f97da4e10aecd50a8d9a9a43c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 6.
I20260925 03:05:08.266286 25196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e92f377a79354a6ebb4eec67303c13e3" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
W20260925 03:05:08.217168 24585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.4:43979, user_credentials={real_user=slave}} blocked reactor thread for 53148.5us
I20260925 03:05:08.271687 24547 raft_consensus.cc:3037] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Advancing to term 8
I20260925 03:05:08.275506 24511 leader_election.cc:302] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:05:08.293563 24619 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f22ddd7d1c444dc58f45a1fb0b3e02bc"
dest_uuid: "eeab93abfdb141a895002ec9f312296c"
mode: GRACEFUL
new_leader_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
from {username='slave'} at 127.0.0.1:44570
I20260925 03:05:08.293684 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ac285175464fbda67211d9574db415" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:08.294162 24619 raft_consensus.cc:595] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c [term 7 LEADER]: Received request to transfer leadership to TS a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:05:08.285609 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d89396576884f7c8901dba54f783947" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:08.229070 27255 raft_consensus.cc:686] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 LEADER]: Becoming Leader. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:05:08.236325 26762 raft_consensus.cc:504] T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:08.346998 24621 raft_consensus.cc:1226] T 21c43af5e6844fdaacc8c625fa01fd8d P eeab93abfdb141a895002ec9f312296c [term 6 LEADER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 5. Current term is 6. Ops: []
W20260925 03:05:08.364125 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 158490us
I20260925 03:05:08.371819 24437 leader_election.cc:302] T 3d89396576884f7c8901dba54f783947 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:08.357014 24616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7691c6b475428c82d6181d693d26f3" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:05:08.349413 24618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1414dcfe53f64a78ae413531eb8ec117" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:08.312215 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e92f377a79354a6ebb4eec67303c13e3" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:08.378221 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48ca6ccd98194cf2af7b5278d15bf4ea" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:08.378981 24469 raft_consensus.cc:2454] T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 5.
I20260925 03:05:08.261684 27339 raft_consensus.cc:3037] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:05:08.386787 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35992316cb8d4c028333388eff02ecde" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:05:08.392519 24511 leader_election.cc:302] T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 1bc1f2e62ab44af995165bd5116a9d43, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:08.393453 24548 raft_consensus.cc:3037] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Advancing to term 8
I20260925 03:05:08.393810 27299 raft_consensus.cc:2726] T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260925 03:05:08.390851 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7aa47c48404465b94a36fd819fe3c7a" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:08.399063 24471 raft_consensus.cc:3037] T a7aa47c48404465b94a36fd819fe3c7a P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:05:08.398528 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48ca6ccd98194cf2af7b5278d15bf4ea" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:08.401816 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c23337a7cb74184a0e57b7095cb6a11" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:08.402456 24464 raft_consensus.cc:3037] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Advancing to term 8
I20260925 03:05:08.347105 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e5819c15eac4544a57e5ee6c9d4f117" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:05:08.400803 24542 raft_consensus.cc:2454] T 48ca6ccd98194cf2af7b5278d15bf4ea P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 5.
I20260925 03:05:08.403976 24368 catalog_manager.cc:5495] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 reported cstate change: term changed from 3 to 6, leader changed from 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1) to eeab93abfdb141a895002ec9f312296c (127.22.193.3). New cstate: current_term: 6 leader_uuid: "eeab93abfdb141a895002ec9f312296c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } }
I20260925 03:05:08.406698 24467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25102ba08e4e49388f6176badcd10a25" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:08.407364 24467 raft_consensus.cc:3037] T 25102ba08e4e49388f6176badcd10a25 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:05:08.408098 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b5df303cd6a4d59aca5c18bbbff24ed" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:08.350096 27232 raft_consensus.cc:482] T 25102ba08e4e49388f6176badcd10a25 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:08.350869 27282 raft_consensus.cc:3037] T c88cbd39e01f44628a094af516be2833 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Advancing to term 8
I20260925 03:05:08.415128 27232 raft_consensus.cc:504] T 25102ba08e4e49388f6176badcd10a25 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:08.416463 27190 consensus_queue.cc:237] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:08.417610 27077 raft_consensus.cc:3032] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 LEADER]: Stepping down as leader of term 7
I20260925 03:05:08.417997 27077 raft_consensus.cc:729] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:05:08.391451 26946 consensus_queue.cc:1035] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.029s
I20260925 03:05:08.287361 27291 raft_consensus.cc:729] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c [term 7 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:05:08.420945 27291 consensus_queue.cc:260] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:08.311856 27273 raft_consensus.cc:2781] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader election won for term 7
I20260925 03:05:08.421254 27077 consensus_queue.cc:260] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:08.422128 27273 raft_consensus.cc:686] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 LEADER]: Becoming Leader. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:05:08.422454 27077 raft_consensus.cc:3037] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Advancing to term 8
I20260925 03:05:08.423039 27273 consensus_queue.cc:237] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:08.312341 27230 raft_consensus.cc:482] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:08.426051 26762 leader_election.cc:288] T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 7 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:05:08.425823 27230 raft_consensus.cc:504] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:08.296838 27022 raft_consensus.cc:482] T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:05:08.391000 27269 raft_consensus.cc:686] T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 LEADER]: Becoming Leader. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:05:08.295126 27323 raft_consensus.cc:2781] T 43366a0a2f9d4344bf4e37ffd290ef8a P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260925 03:05:08.376019 27304 raft_consensus.cc:2726] T 3d89396576884f7c8901dba54f783947 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260925 03:05:08.425848 24368 catalog_manager.cc:5495] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 reported cstate change: term changed from 5 to 7, leader changed from eeab93abfdb141a895002ec9f312296c (127.22.193.3) to 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1). New cstate: current_term: 7 leader_uuid: "869b0a8575b54d08ae2817fad5dd42d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: HEALTHY } } }
I20260925 03:05:08.420032 24464 raft_consensus.cc:2454] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 8.
I20260925 03:05:08.433688 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abd5b8dc1b7e49e3aa9af900d73ff0e7" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:08.428192 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f68804eb5a244d87bb395d50683c6012" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:08.430946 27323 raft_consensus.cc:482] T 43366a0a2f9d4344bf4e37ffd290ef8a P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:05:08.429499 24471 raft_consensus.cc:2454] T a7aa47c48404465b94a36fd819fe3c7a P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 7.
I20260925 03:05:08.436527 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15861cd28e354598a3c6bfc4337c34bb" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:08.403195 24548 raft_consensus.cc:2454] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 8.
I20260925 03:05:08.436939 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5332a5d329493991dff994e70ea92c" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:08.438362 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4ebd2b043024a548005c8b6b00b0449" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:08.440303 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c88cbd39e01f44628a094af516be2833" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:08.478652 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e725b82985746b39c0cc8f671c95353" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
W20260925 03:05:08.479770 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 47716us
I20260925 03:05:08.307523 24541 raft_consensus.cc:2379] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1bc1f2e62ab44af995165bd5116a9d43 in current term 9: Already voted for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in this term.
I20260925 03:05:08.500414 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b627ecb4a5384ef392aa695e06599c39" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:08.501691 24469 raft_consensus.cc:3037] T b627ecb4a5384ef392aa695e06599c39 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:05:08.505635 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6b6ec010caf4689b8c90449764f5cf9" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:08.506505 24473 raft_consensus.cc:2379] T e6b6ec010caf4689b8c90449764f5cf9 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 7: Already voted for candidate 1bc1f2e62ab44af995165bd5116a9d43 in this term.
I20260925 03:05:08.507300 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37a157996e944db2906110d87ecb0c54" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
W20260925 03:05:08.508611 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 105194us
I20260925 03:05:08.510519 24588 leader_election.cc:302] T 48ca6ccd98194cf2af7b5278d15bf4ea P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:05:08.518414 24548 raft_consensus.cc:1226] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20260925 03:05:08.522331 24466 raft_consensus.cc:1261] T 293ef69caf514ac98510a68495f54eda P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 5 index: 8. Preceding OpId from leader: term: 6 index: 9. (index mismatch)
I20260925 03:05:08.431198 27022 raft_consensus.cc:3037] T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Advancing to term 8
I20260925 03:05:08.514510 24368 catalog_manager.cc:5495] T e6b6ec010caf4689b8c90449764f5cf9 P 869b0a8575b54d08ae2817fad5dd42d5 reported cstate change: term changed from 3 to 4, leader changed from 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) to 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1). New cstate: current_term: 4 leader_uuid: "869b0a8575b54d08ae2817fad5dd42d5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260925 03:05:08.527969 25163 leader_election.cc:302] T 29637f5a4a504ce6ba29033ccd0cd716 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:08.532663 24514 leader_election.cc:302] T 0b5df303cd6a4d59aca5c18bbbff24ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:08.477662 27323 raft_consensus.cc:3037] T 43366a0a2f9d4344bf4e37ffd290ef8a P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:05:08.542102 24514 leader_election.cc:302] T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:05:08.542512 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:08.543350 24473 raft_consensus.cc:3037] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Advancing to term 9
I20260925 03:05:08.544655 24514 leader_election.cc:302] T a7aa47c48404465b94a36fd819fe3c7a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:05:08.513135 27279 raft_consensus.cc:482] T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:08.554328 27279 raft_consensus.cc:504] T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:08.556535 27230 leader_election.cc:288] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
W20260925 03:05:08.557150 27348 log.cc:924] Time spent T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.192s user 0.000s sys 0.000s
I20260925 03:05:08.559229 27269 consensus_queue.cc:237] T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:08.569916 26762 consensus_queue.cc:1046] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260925 03:05:08.556388 27279 leader_election.cc:288] T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:08.522119 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442d4a29b1d743599f4b6b4871e26668" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:08.575958 24469 raft_consensus.cc:2454] T b627ecb4a5384ef392aa695e06599c39 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 7.
I20260925 03:05:08.582406 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d89396576884f7c8901dba54f783947" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:08.583030 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e5819c15eac4544a57e5ee6c9d4f117" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:08.522702 24540 raft_consensus.cc:1226] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 LEADER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 5. Current term is 6. Ops: []
I20260925 03:05:08.576793 24464 raft_consensus.cc:2454] T c88cbd39e01f44628a094af516be2833 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 8.
I20260925 03:05:08.520458 27304 raft_consensus.cc:482] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:08.531421 27052 raft_consensus.cc:2781] T 48ca6ccd98194cf2af7b5278d15bf4ea P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260925 03:05:08.593894 23300 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260925 03:05:08.599676 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74443a0a1dc34d88a2f42e3b722db405" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:08.601545 24471 raft_consensus.cc:2379] T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 6: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:05:08.602676 23300 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260925 03:05:08.603420 23300 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260925 03:05:08.604642 24468 raft_consensus.cc:2454] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 7.
I20260925 03:05:08.538812 27268 raft_consensus.cc:2781] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Leader election won for term 9
I20260925 03:05:08.605873 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bb46bce3564841b7674e6a693bd2e6" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:08.537907 27322 raft_consensus.cc:2726] T 0b5df303cd6a4d59aca5c18bbbff24ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260925 03:05:08.629555 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7691c6b475428c82d6181d693d26f3" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:08.630733 24468 raft_consensus.cc:3037] T 11bb46bce3564841b7674e6a693bd2e6 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:05:08.603981 24467 raft_consensus.cc:2454] T 25102ba08e4e49388f6176badcd10a25 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 6.
I20260925 03:05:08.632741 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ac285175464fbda67211d9574db415" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:08.408872 27301 consensus_queue.cc:237] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:08.579936 27341 raft_consensus.cc:2726] T 29637f5a4a504ce6ba29033ccd0cd716 P 1bc1f2e62ab44af995165bd5116a9d43 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260925 03:05:08.580107 27299 raft_consensus.cc:2781] T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Leader election won for term 8
I20260925 03:05:08.587219 26800 raft_consensus.cc:2781] T a7aa47c48404465b94a36fd819fe3c7a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Leader election won for term 7
I20260925 03:05:08.639463 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60d45b7d467d4e1ab92367f6d7d52b15" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:08.640101 24471 raft_consensus.cc:3037] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Advancing to term 9
I20260925 03:05:08.588129 27232 leader_election.cc:288] T 25102ba08e4e49388f6176badcd10a25 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:05:08.642907 24467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1414dcfe53f64a78ae413531eb8ec117" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:08.651254 24468 raft_consensus.cc:2454] T 11bb46bce3564841b7674e6a693bd2e6 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 4.
I20260925 03:05:08.638727 24368 catalog_manager.cc:5495] T 0be4e0aa984f4f2b81a58e382e434ad5 P 869b0a8575b54d08ae2817fad5dd42d5 reported cstate change: term changed from 3 to 4, leader changed from eeab93abfdb141a895002ec9f312296c (127.22.193.3) to 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1). New cstate: current_term: 4 leader_uuid: "869b0a8575b54d08ae2817fad5dd42d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: HEALTHY } } }
I20260925 03:05:08.590471 27041 raft_consensus.cc:3032] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c [term 7 LEADER]: Stepping down as leader of term 7
I20260925 03:05:08.630996 27052 raft_consensus.cc:482] T 48ca6ccd98194cf2af7b5278d15bf4ea P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:05:08.669765 24467 raft_consensus.cc:1226] T 21c43af5e6844fdaacc8c625fa01fd8d P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 5. Current term is 6. Ops: []
I20260925 03:05:08.677165 24545 raft_consensus.cc:1226] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 LEADER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 6. Current term is 7. Ops: []
I20260925 03:05:08.677500 24468 raft_consensus.cc:1226] T 9e8af2641e124f9589c5ccfdac785f63 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 6. Current term is 7. Ops: [6.6-6.6]
I20260925 03:05:08.686174 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f1fac028a144058b92331c0d9b44ebf" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:08.692266 24473 raft_consensus.cc:2454] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 9.
I20260925 03:05:08.636207 27279 raft_consensus.cc:482] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:08.641923 27268 raft_consensus.cc:686] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 LEADER]: Becoming Leader. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:05:08.659448 27041 raft_consensus.cc:729] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c [term 7 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:05:08.587536 27255 consensus_queue.cc:237] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:08.703100 24368 catalog_manager.cc:5495] T c85f5d1d3f004cf5817bb594fd8e6469 P 869b0a8575b54d08ae2817fad5dd42d5 reported cstate change: term changed from 3 to 4, leader changed from 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1) to eeab93abfdb141a895002ec9f312296c (127.22.193.3). New cstate: current_term: 4 leader_uuid: "eeab93abfdb141a895002ec9f312296c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } }
I20260925 03:05:08.678431 27052 raft_consensus.cc:3037] T 48ca6ccd98194cf2af7b5278d15bf4ea P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:05:08.670593 27322 raft_consensus.cc:482] T 420163a6cdce412ebc21c25fe3835638 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:08.670794 26800 raft_consensus.cc:686] T a7aa47c48404465b94a36fd819fe3c7a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 LEADER]: Becoming Leader. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:05:08.696439 27353 raft_consensus.cc:482] T a7aa47c48404465b94a36fd819fe3c7a P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260925 03:05:08.729547 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 207673us
I20260925 03:05:08.628064 27304 raft_consensus.cc:504] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:08.737159 24368 catalog_manager.cc:5495] T 7993f9c3e2a44f408bbf5d1e395e2306 P 869b0a8575b54d08ae2817fad5dd42d5 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "eeab93abfdb141a895002ec9f312296c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } }
I20260925 03:05:08.742339 24368 catalog_manager.cc:5495] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5 reported cstate change: term changed from 1 to 3, leader changed from a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2) to 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1). New cstate: current_term: 3 leader_uuid: "869b0a8575b54d08ae2817fad5dd42d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: HEALTHY } } }
I20260925 03:05:08.746662 24368 catalog_manager.cc:5495] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 reported cstate change: term changed from 2 to 8. New cstate: current_term: 8 leader_uuid: "869b0a8575b54d08ae2817fad5dd42d5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260925 03:05:08.718910 24471 raft_consensus.cc:2454] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 9.
I20260925 03:05:08.734277 27267 raft_consensus.cc:482] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:08.804817 24586 leader_election.cc:302] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:05:08.725332 27279 raft_consensus.cc:504] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:05:08.803925 27349 log.cc:924] Time spent T 04e068a0f80941dc9138cee063f53bab P 1bc1f2e62ab44af995165bd5116a9d43: Append to log took a long time: real 0.154s user 0.001s sys 0.001s
I20260925 03:05:08.732838 27322 raft_consensus.cc:504] T 420163a6cdce412ebc21c25fe3835638 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:08.739110 27353 raft_consensus.cc:504] T a7aa47c48404465b94a36fd819fe3c7a P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:05:08.855530 25162 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 321029us
I20260925 03:05:08.757619 27243 consensus_queue.cc:1035] T 04e068a0f80941dc9138cee063f53bab P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.052s
I20260925 03:05:08.766304 27253 consensus_queue.cc:1035] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.047s
W20260925 03:05:08.900166 24547 consensus_meta.cc:287] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent flushing consensus metadata: real 0.603s user 0.007s sys 0.010s
I20260925 03:05:08.900642 24547 raft_consensus.cc:2454] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 8.
I20260925 03:05:08.896039 27232 raft_consensus.cc:482] T 11bb46bce3564841b7674e6a693bd2e6 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:08.808477 27267 raft_consensus.cc:504] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:08.802336 27041 consensus_queue.cc:260] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:08.918481 27299 raft_consensus.cc:686] T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 LEADER]: Becoming Leader. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
W20260925 03:05:08.949034 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 437685us
W20260925 03:05:08.953444 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 148691us
I20260925 03:05:08.926216 27252 raft_consensus.cc:482] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:08.925549 27364 raft_consensus.cc:2781] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Leader election won for term 8
W20260925 03:05:08.951692 27351 log.cc:924] Time spent T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.255s user 0.001s sys 0.000s
W20260925 03:05:08.964380 27355 log.cc:924] Time spent T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.244s user 0.001s sys 0.000s
W20260925 03:05:08.969913 27350 log.cc:924] Time spent T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.244s user 0.000s sys 0.001s
I20260925 03:05:08.899129 27268 consensus_queue.cc:237] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.6, Last appended by leader: 6, Current term: 9, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:08.878465 26800 consensus_queue.cc:237] T a7aa47c48404465b94a36fd819fe3c7a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:08.945267 27232 raft_consensus.cc:504] T 11bb46bce3564841b7674e6a693bd2e6 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:05:09.008330 27360 log.cc:924] Time spent T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.077s user 0.001s sys 0.000s
W20260925 03:05:08.970888 27339 consensus_meta.cc:287] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43: Time spent flushing consensus metadata: real 0.559s user 0.010s sys 0.000s
W20260925 03:05:09.012662 27365 log.cc:924] Time spent T e6b6ec010caf4689b8c90449764f5cf9 P 1bc1f2e62ab44af995165bd5116a9d43: Append to log took a long time: real 0.063s user 0.000s sys 0.000s
I20260925 03:05:09.012982 27323 raft_consensus.cc:504] T 43366a0a2f9d4344bf4e37ffd290ef8a P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:05:09.013891 27358 log.cc:924] Time spent T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.209s user 0.000s sys 0.001s
W20260925 03:05:09.015064 27359 log.cc:924] Time spent T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5: Append to log took a long time: real 0.061s user 0.000s sys 0.000s
I20260925 03:05:08.972632 27022 raft_consensus.cc:504] T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:09.016846 27022 leader_election.cc:288] T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 8 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:05:08.978633 27304 leader_election.cc:288] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
W20260925 03:05:09.029970 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 75742us
I20260925 03:05:08.963085 27252 raft_consensus.cc:504] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
W20260925 03:05:09.032171 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 487616us
I20260925 03:05:09.015100 27323 leader_election.cc:288] T 43366a0a2f9d4344bf4e37ffd290ef8a P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:05:09.044960 27279 leader_election.cc:288] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
W20260925 03:05:09.108291 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 77614.3us
I20260925 03:05:09.023289 27339 raft_consensus.cc:504] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:09.118257 24586 leader_election.cc:302] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:05:09.028148 27299 consensus_queue.cc:237] T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:09.104238 27322 leader_election.cc:288] T 420163a6cdce412ebc21c25fe3835638 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:09.106364 27353 leader_election.cc:288] T a7aa47c48404465b94a36fd819fe3c7a P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:05:09.127890 25199 raft_consensus.cc:1226] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43 [term 7 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 6. Current term is 7. Ops: []
I20260925 03:05:09.129053 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c23337a7cb74184a0e57b7095cb6a11" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:09.130673 24473 raft_consensus.cc:1261] T 21c43af5e6844fdaacc8c625fa01fd8d P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260925 03:05:09.131181 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be4e0aa984f4f2b81a58e382e434ad5" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:09.130712 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25102ba08e4e49388f6176badcd10a25" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:09.131850 24472 raft_consensus.cc:3037] T 0be4e0aa984f4f2b81a58e382e434ad5 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:05:09.132170 24464 raft_consensus.cc:2454] T 25102ba08e4e49388f6176badcd10a25 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 6.
I20260925 03:05:09.129760 24471 raft_consensus.cc:2454] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 8.
I20260925 03:05:09.144424 27252 leader_election.cc:288] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:05:09.145347 27232 leader_election.cc:288] T 11bb46bce3564841b7674e6a693bd2e6 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:05:09.146929 24470 raft_consensus.cc:1261] T c88cbd39e01f44628a094af516be2833 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260925 03:05:09.148437 24473 raft_consensus.cc:1261] T b627ecb4a5384ef392aa695e06599c39 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260925 03:05:09.150166 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e8af2641e124f9589c5ccfdac785f63" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:09.150825 24471 raft_consensus.cc:2454] T 9e8af2641e124f9589c5ccfdac785f63 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 7.
I20260925 03:05:09.155212 24464 raft_consensus.cc:1261] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 5. (term mismatch)
I20260925 03:05:09.155613 24464 pending_rounds.cc:76] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5: Aborting all ops after (but not including) 4
I20260925 03:05:09.155804 24464 pending_rounds.cc:97] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5: Aborting uncommitted NO_OP operation due to leader change: 8.5
I20260925 03:05:09.156018 24464 raft_consensus.cc:2866] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260925 03:05:09.156114 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9c33fd100ff44c1a3247319a324a044" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:09.157415 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43366a0a2f9d4344bf4e37ffd290ef8a" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:09.157974 24470 raft_consensus.cc:3037] T 43366a0a2f9d4344bf4e37ffd290ef8a P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Advancing to term 5
W20260925 03:05:09.158752 27272 consensus_meta.cc:287] T ceab6b0fccd54e909a64f66fe81a9a1f P 869b0a8575b54d08ae2817fad5dd42d5: Time spent flushing consensus metadata: real 1.079s user 0.009s sys 0.001s
I20260925 03:05:09.165073 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b8e0d260a884ecdb20f79de46c36b19" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:09.165872 24468 raft_consensus.cc:2379] T 7b8e0d260a884ecdb20f79de46c36b19 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 7: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:05:09.169123 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c23337a7cb74184a0e57b7095cb6a11" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:09.170306 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be4e0aa984f4f2b81a58e382e434ad5" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:05:09.170791 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25102ba08e4e49388f6176badcd10a25" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:09.171468 24540 raft_consensus.cc:2454] T 25102ba08e4e49388f6176badcd10a25 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 6.
I20260925 03:05:09.170943 24545 raft_consensus.cc:3037] T 0be4e0aa984f4f2b81a58e382e434ad5 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:05:09.173840 24465 raft_consensus.cc:2396] T ceab6b0fccd54e909a64f66fe81a9a1f P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c for term 8 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 3.
I20260925 03:05:09.157428 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7aa47c48404465b94a36fd819fe3c7a" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:09.180781 24464 raft_consensus.cc:2454] T a7aa47c48404465b94a36fd819fe3c7a P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 7.
I20260925 03:05:09.188987 24467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "420163a6cdce412ebc21c25fe3835638" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
W20260925 03:05:09.192399 24544 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.505s user 0.027s sys 0.008s
W20260925 03:05:09.193001 24511 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 78226.9us
I20260925 03:05:09.194329 27325 raft_consensus.cc:482] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:09.197137 24547 raft_consensus.cc:1261] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260925 03:05:09.168663 24472 raft_consensus.cc:2454] T 0be4e0aa984f4f2b81a58e382e434ad5 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 7.
W20260925 03:05:09.200105 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 82718.1us
I20260925 03:05:09.202312 24541 raft_consensus.cc:1261] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Refusing update from remote peer 1bc1f2e62ab44af995165bd5116a9d43: Log matching property violated. Preceding OpId in replica: term: 5 index: 8. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20260925 03:05:09.202085 24514 leader_election.cc:302] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:09.203792 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43366a0a2f9d4344bf4e37ffd290ef8a" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:05:09.204440 24540 raft_consensus.cc:3037] T 43366a0a2f9d4344bf4e37ffd290ef8a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:05:09.205989 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9c33fd100ff44c1a3247319a324a044" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:05:09.206707 24547 raft_consensus.cc:3037] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Advancing to term 8
I20260925 03:05:09.211535 24547 raft_consensus.cc:2454] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 8.
I20260925 03:05:09.206441 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7aa47c48404465b94a36fd819fe3c7a" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:09.213238 24547 raft_consensus.cc:1261] T 5278a60f97da4e10aecd50a8d9a9a43c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260925 03:05:09.214591 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5f9b503b0ab43999d582015a367441e" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:09.192142 27273 raft_consensus.cc:482] T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1bc1f2e62ab44af995165bd5116a9d43)
I20260925 03:05:09.194823 27325 raft_consensus.cc:504] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:09.216833 24543 raft_consensus.cc:1226] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 LEADER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 6. Current term is 7. Ops: []
I20260925 03:05:09.227201 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e8af2641e124f9589c5ccfdac785f63" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:09.227883 24621 raft_consensus.cc:1261] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260925 03:05:09.232965 24467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bb46bce3564841b7674e6a693bd2e6" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:09.239423 24467 raft_consensus.cc:2454] T 11bb46bce3564841b7674e6a693bd2e6 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 4.
I20260925 03:05:09.233753 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c81a1b3b616492ab6c964f45af36077" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:09.200477 27383 raft_consensus.cc:482] T 550118bc1a0641ce9803bdbf43f2069f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:09.241703 24619 raft_consensus.cc:2454] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 7.
I20260925 03:05:09.194895 27279 consensus_queue.cc:1046] T 21c43af5e6844fdaacc8c625fa01fd8d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.031s
I20260925 03:05:09.212270 26946 consensus_queue.cc:1035] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.036s
I20260925 03:05:09.237264 24614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b8e0d260a884ecdb20f79de46c36b19" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:09.199658 27369 raft_consensus.cc:482] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:05:09.238085 24586 leader_election.cc:302] T 25102ba08e4e49388f6176badcd10a25 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:05:09.221345 27385 raft_consensus.cc:2781] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Leader election won for term 8
I20260925 03:05:09.231865 24540 raft_consensus.cc:2454] T 43366a0a2f9d4344bf4e37ffd290ef8a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 5.
I20260925 03:05:09.243021 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "420163a6cdce412ebc21c25fe3835638" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:09.246007 24539 raft_consensus.cc:1261] T 37a157996e944db2906110d87ecb0c54 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260925 03:05:09.240651 26900 raft_consensus.cc:3032] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c [term 8 LEADER]: Stepping down as leader of term 8
I20260925 03:05:09.192813 27304 raft_consensus.cc:482] T 442d4a29b1d743599f4b6b4871e26668 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:09.235342 27252 raft_consensus.cc:482] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:05:09.260097 26900 raft_consensus.cc:729] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c [term 8 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:05:09.247299 27383 raft_consensus.cc:504] T 550118bc1a0641ce9803bdbf43f2069f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:09.259328 24614 raft_consensus.cc:2454] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 7.
I20260925 03:05:09.236099 27273 raft_consensus.cc:504] T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:09.259683 27384 raft_consensus.cc:2781] T 25102ba08e4e49388f6176badcd10a25 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260925 03:05:09.278491 24586 leader_election.cc:302] T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:05:09.263705 27385 raft_consensus.cc:686] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [term 8 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:05:09.266806 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bb46bce3564841b7674e6a693bd2e6" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:09.255553 27279 consensus_queue.cc:1225] T 21c43af5e6844fdaacc8c625fa01fd8d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Peer eeab93abfdb141a895002ec9f312296c log is divergent from this leader: its last log entry 6.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260925 03:05:09.268829 24539 raft_consensus.cc:1226] T 8e5819c15eac4544a57e5ee6c9d4f117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 7. Current term is 8. Ops: []
I20260925 03:05:09.265098 27339 leader_election.cc:288] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 7 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:05:09.263417 27249 consensus_queue.cc:1035] T 37a157996e944db2906110d87ecb0c54 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.024s
I20260925 03:05:09.290405 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "293ef69caf514ac98510a68495f54eda" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:09.291702 24542 raft_consensus.cc:1226] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 7. Current term is 8. Ops: [7.4-7.4]
W20260925 03:05:09.301529 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 98717.2us
I20260925 03:05:09.311721 24464 raft_consensus.cc:3037] T 293ef69caf514ac98510a68495f54eda P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:05:09.296039 24586 leader_election.cc:302] T 43366a0a2f9d4344bf4e37ffd290ef8a P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
W20260925 03:05:09.316138 27388 log.cc:924] Time spent T abd5b8dc1b7e49e3aa9af900d73ff0e7 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.075s user 0.000s sys 0.000s
I20260925 03:05:09.285554 27243 raft_consensus.cc:2781] T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Leader election won for term 8
I20260925 03:05:09.309190 27333 consensus_queue.cc:1035] T 5278a60f97da4e10aecd50a8d9a9a43c P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.009s
I20260925 03:05:09.309523 24467 raft_consensus.cc:1261] T e6b6ec010caf4689b8c90449764f5cf9 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Refusing update from remote peer 1bc1f2e62ab44af995165bd5116a9d43: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20260925 03:05:09.282578 27385 consensus_queue.cc:237] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:09.320851 26900 consensus_queue.cc:260] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:09.322953 27383 leader_election.cc:288] T 550118bc1a0641ce9803bdbf43f2069f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
W20260925 03:05:09.323762 27052 consensus_meta.cc:287] T 48ca6ccd98194cf2af7b5278d15bf4ea P eeab93abfdb141a895002ec9f312296c: Time spent flushing consensus metadata: real 0.590s user 0.011s sys 0.000s
I20260925 03:05:09.324160 27052 raft_consensus.cc:504] T 48ca6ccd98194cf2af7b5278d15bf4ea P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:09.325083 26946 consensus_queue.cc:1046] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260925 03:05:09.325906 27346 raft_consensus.cc:3032] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 LEADER]: Stepping down as leader of term 7
I20260925 03:05:09.326308 27346 raft_consensus.cc:729] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:05:09.327109 27346 consensus_queue.cc:260] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:09.255738 27369 raft_consensus.cc:504] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:09.267192 27252 raft_consensus.cc:504] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:09.277376 27384 raft_consensus.cc:482] T 25102ba08e4e49388f6176badcd10a25 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:05:09.328653 27346 raft_consensus.cc:3037] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Advancing to term 8
I20260925 03:05:09.334208 27254 consensus_queue.cc:1035] T e6b6ec010caf4689b8c90449764f5cf9 P 1bc1f2e62ab44af995165bd5116a9d43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260925 03:05:09.330410 27369 leader_election.cc:288] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:09.331282 27252 leader_election.cc:288] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:05:09.331461 27384 raft_consensus.cc:3037] T 25102ba08e4e49388f6176badcd10a25 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:05:09.265250 27304 raft_consensus.cc:504] T 442d4a29b1d743599f4b6b4871e26668 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:05:09.341467 27347 log.cc:924] Time spent T 21c43af5e6844fdaacc8c625fa01fd8d P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.704s user 0.003s sys 0.021s
I20260925 03:05:09.341740 27304 leader_election.cc:288] T 442d4a29b1d743599f4b6b4871e26668 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:09.343034 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442d4a29b1d743599f4b6b4871e26668" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:09.343902 24539 raft_consensus.cc:2379] T 442d4a29b1d743599f4b6b4871e26668 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 in current term 8: Already voted for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in this term.
I20260925 03:05:09.326097 27052 leader_election.cc:288] T 48ca6ccd98194cf2af7b5278d15bf4ea P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 6 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:05:09.347893 27267 leader_election.cc:288] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:09.348129 27052 raft_consensus.cc:482] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:09.348593 27052 raft_consensus.cc:504] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:09.327174 27273 leader_election.cc:288] T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:05:09.354108 25199 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48424 (request call id 271) took 1083 ms. Trace:
I20260925 03:05:09.354512 25199 rpcz_store.cc:276] 0925 03:05:08.270831 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:05:08.270988 (+ 157us) service_pool.cc:227] Handling call
0925 03:05:08.271338 (+ 350us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:05:09.128399 (+857061us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:05:09.353880 (+225481us) spinlock_profiling.cc:243] Waited 856 ms on lock 0x7b54006db854. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d0f51 00007f43764cf99e 00007f437b4112e9 00007f4374af7684 00007f4374af7612 00007f4374af7541 00007f4374af74bc 00007f4374af6772 00007f43748fa94c 00007f43748f9d86
0925 03:05:09.354050 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1969410432}
I20260925 03:05:09.355105 27243 raft_consensus.cc:686] T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c [term 8 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:05:09.358767 24464 raft_consensus.cc:2454] T 293ef69caf514ac98510a68495f54eda P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 7.
I20260925 03:05:09.361919 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7691c6b475428c82d6181d693d26f3" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:09.297869 24470 raft_consensus.cc:2454] T 43366a0a2f9d4344bf4e37ffd290ef8a P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 5.
I20260925 03:05:09.374564 24465 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 1367) took 1360 ms. Trace:
I20260925 03:05:09.375065 24465 rpcz_store.cc:276] 0925 03:05:08.014133 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:05:08.022693 (+ 8560us) service_pool.cc:227] Handling call
0925 03:05:09.374299 (+1351606us) spinlock_profiling.cc:243] Waited 1.11 s on lock 0x7b54003f7d54. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d42ae 00007f437b411c27 00007f4374af9c04 00007f4374af9b92 00007f4374af9ac1 00007f4374af9a3c 00007f4374af8cf2 00007f43748fa94c 00007f43748f9d86 00007f43748f972f
0925 03:05:09.374497 (+ 198us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2553598208}
I20260925 03:05:09.376947 24514 leader_election.cc:302] T b4ebd2b043024a548005c8b6b00b0449 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:05:09.378221 24514 leader_election.cc:302] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:09.383092 24545 raft_consensus.cc:2454] T 0be4e0aa984f4f2b81a58e382e434ad5 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 7.
I20260925 03:05:09.285173 24547 raft_consensus.cc:2454] T 11bb46bce3564841b7674e6a693bd2e6 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 4.
I20260925 03:05:09.356231 27243 consensus_queue.cc:237] T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:09.385048 27279 raft_consensus.cc:2726] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260925 03:05:09.384222 27369 raft_consensus.cc:2726] T b4ebd2b043024a548005c8b6b00b0449 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260925 03:05:09.356190 27357 log.cc:924] Time spent T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.548s user 0.000s sys 0.008s
I20260925 03:05:09.384232 27190 consensus_queue.cc:1046] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.002s
I20260925 03:05:09.386039 27376 raft_consensus.cc:979] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c: : Instructing follower a04a37fc4aa7460cbfdbeeb4e451e1f5 to start an election
W20260925 03:05:09.382107 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 86156.8us
I20260925 03:05:09.383677 24514 leader_election.cc:302] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:05:09.414109 24586 leader_election.cc:302] T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:05:09.415602 24586 leader_election.cc:302] T 11bb46bce3564841b7674e6a693bd2e6 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:05:09.422847 24514 leader_election.cc:302] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:05:09.426491 24514 leader_election.cc:302] T 37a157996e944db2906110d87ecb0c54 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:09.427148 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48ca6ccd98194cf2af7b5278d15bf4ea" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:05:09.427803 24545 raft_consensus.cc:3037] T 48ca6ccd98194cf2af7b5278d15bf4ea P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:05:09.432145 24543 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 1361) took 1281 ms. Trace:
I20260925 03:05:09.432571 24543 rpcz_store.cc:276] 0925 03:05:08.151140 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:05:08.151589 (+ 449us) service_pool.cc:227] Handling call
0925 03:05:08.151931 (+ 342us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:05:09.240889 (+1088958us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:05:09.431889 (+191000us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b540004ced4. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d0f51 00007f43764cf99e 00007f437b4112e9 00007f4374af7684 00007f4374af7612 00007f4374af7541 00007f4374af74bc 00007f4374af6772 00007f43748fa94c 00007f43748f9d86
0925 03:05:09.432085 (+ 196us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2448652032}
I20260925 03:05:09.385362 27354 raft_consensus.cc:2781] T 43366a0a2f9d4344bf4e37ffd290ef8a P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Leader election won for term 5
I20260925 03:05:09.433537 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abd5b8dc1b7e49e3aa9af900d73ff0e7" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:09.410529 26829 raft_consensus.cc:482] T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:09.427405 27252 raft_consensus.cc:2781] T 11bb46bce3564841b7674e6a693bd2e6 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260925 03:05:09.471458 27259 raft_consensus.cc:482] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1bc1f2e62ab44af995165bd5116a9d43)
I20260925 03:05:09.473914 27009 consensus_queue.cc:1035] T 21c43af5e6844fdaacc8c625fa01fd8d P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260925 03:05:09.474638 27392 raft_consensus.cc:2726] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260925 03:05:09.427528 27190 raft_consensus.cc:2726] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260925 03:05:09.476306 24539 raft_consensus.cc:1226] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 7. Current term is 8. Ops: []
I20260925 03:05:09.476593 24588 leader_election.cc:302] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:05:09.478060 24514 leader_election.cc:302] T 442d4a29b1d743599f4b6b4871e26668 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:09.403393 27376 raft_consensus.cc:1067] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c [term 7 LEADER]: Signalling peer a04a37fc4aa7460cbfdbeeb4e451e1f5 to start an election
I20260925 03:05:09.413368 27052 leader_election.cc:288] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:05:09.360606 27384 raft_consensus.cc:504] T 25102ba08e4e49388f6176badcd10a25 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:09.475859 24545 raft_consensus.cc:2454] T 48ca6ccd98194cf2af7b5278d15bf4ea P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 6.
I20260925 03:05:09.386797 27255 raft_consensus.cc:482] T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:09.424407 27053 raft_consensus.cc:2781] T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Leader election won for term 7
I20260925 03:05:09.499992 24588 leader_election.cc:302] T a7aa47c48404465b94a36fd819fe3c7a P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5
W20260925 03:05:09.501130 25165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 168729us
I20260925 03:05:09.503341 24586 leader_election.cc:302] T 48ca6ccd98194cf2af7b5278d15bf4ea P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:05:09.504144 24514 leader_election.cc:302] T 3d89396576884f7c8901dba54f783947 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:09.493176 27252 raft_consensus.cc:482] T 11bb46bce3564841b7674e6a693bd2e6 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:05:09.500759 27354 raft_consensus.cc:686] T 43366a0a2f9d4344bf4e37ffd290ef8a P eeab93abfdb141a895002ec9f312296c [term 5 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:05:09.420519 27328 consensus_queue.cc:1046] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.081s
I20260925 03:05:09.509582 24541 raft_consensus.cc:3032] T 21c43af5e6844fdaacc8c625fa01fd8d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 LEADER]: Stepping down as leader of term 5
I20260925 03:05:09.510751 27052 raft_consensus.cc:2781] T a7aa47c48404465b94a36fd819fe3c7a P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260925 03:05:09.491542 27273 raft_consensus.cc:482] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
W20260925 03:05:09.505084 24585 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.512s user 0.006s sys 0.010s
W20260925 03:05:09.526273 24585 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.533s user 0.006s sys 0.010s
I20260925 03:05:09.527140 24467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5aec38e66a947c681e6c2a72b05e6ca" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:09.515349 24514 leader_election.cc:302] T 8e5819c15eac4544a57e5ee6c9d4f117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:09.518504 24541 raft_consensus.cc:729] T 21c43af5e6844fdaacc8c625fa01fd8d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:05:09.497130 27301 raft_consensus.cc:2726] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260925 03:05:09.521612 25165 leader_election.cc:302] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5; no voters:
I20260925 03:05:09.499018 26811 raft_consensus.cc:2726] T 37a157996e944db2906110d87ecb0c54 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260925 03:05:09.539733 25199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1e85248d3b04b979dd080caeeb02fdb" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:05:09.540177 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b8e0d260a884ecdb20f79de46c36b19" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:09.541426 24468 raft_consensus.cc:2454] T 7b8e0d260a884ecdb20f79de46c36b19 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 7.
I20260925 03:05:09.528872 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abd5b8dc1b7e49e3aa9af900d73ff0e7" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:09.542461 24472 raft_consensus.cc:2454] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 6.
I20260925 03:05:09.543471 24616 raft_consensus.cc:1261] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260925 03:05:09.531170 24514 leader_election.cc:302] T 25102ba08e4e49388f6176badcd10a25 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:05:09.526216 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48ca6ccd98194cf2af7b5278d15bf4ea" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:09.545928 24470 raft_consensus.cc:3037] T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:05:09.505851 27022 raft_consensus.cc:2781] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260925 03:05:09.546666 24620 raft_consensus.cc:1226] T 21c43af5e6844fdaacc8c625fa01fd8d P eeab93abfdb141a895002ec9f312296c [term 6 LEADER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260925 03:05:09.548717 24621 raft_consensus.cc:1261] T c88cbd39e01f44628a094af516be2833 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260925 03:05:09.532900 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06f4a7eaf6534e4986c9fd35911038e4" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:09.552481 24619 raft_consensus.cc:2454] T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 8.
I20260925 03:05:09.530047 24541 consensus_queue.cc:260] T 21c43af5e6844fdaacc8c625fa01fd8d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:09.553532 24541 raft_consensus.cc:3037] T 21c43af5e6844fdaacc8c625fa01fd8d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:05:09.501163 27364 raft_consensus.cc:686] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [term 8 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:05:09.554582 27325 leader_election.cc:288] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:05:09.555020 27204 raft_consensus.cc:2781] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43 [term 7 FOLLOWER]: Leader election won for term 7
I20260925 03:05:09.539325 27126 raft_consensus.cc:2726] T 8e5819c15eac4544a57e5ee6c9d4f117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260925 03:05:09.526232 27267 raft_consensus.cc:482] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:09.556353 24467 raft_consensus.cc:1226] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 8. Current term is 9. Ops: []
I20260925 03:05:09.553575 24470 raft_consensus.cc:2454] T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 6.
I20260925 03:05:09.519615 27368 raft_consensus.cc:2726] T 3d89396576884f7c8901dba54f783947 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260925 03:05:09.514046 26800 raft_consensus.cc:2726] T 442d4a29b1d743599f4b6b4871e26668 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260925 03:05:09.518465 27252 raft_consensus.cc:3037] T 11bb46bce3564841b7674e6a693bd2e6 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:05:09.566188 27041 raft_consensus.cc:3037] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Advancing to term 8
I20260925 03:05:09.493814 26829 raft_consensus.cc:504] T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:09.496599 27259 raft_consensus.cc:504] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:09.569693 24465 raft_consensus.cc:1261] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260925 03:05:09.584283 24588 leader_election.cc:302] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:05:09.526551 27052 raft_consensus.cc:482] T a7aa47c48404465b94a36fd819fe3c7a P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:05:09.585172 25198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c81a1b3b616492ab6c964f45af36077" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:05:09.582525 24541 raft_consensus.cc:1261] T 21c43af5e6844fdaacc8c625fa01fd8d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 4. (term mismatch)
I20260925 03:05:09.601313 24541 pending_rounds.cc:76] T 21c43af5e6844fdaacc8c625fa01fd8d P a04a37fc4aa7460cbfdbeeb4e451e1f5: Aborting all ops after (but not including) 3
I20260925 03:05:09.601550 24541 pending_rounds.cc:97] T 21c43af5e6844fdaacc8c625fa01fd8d P a04a37fc4aa7460cbfdbeeb4e451e1f5: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20260925 03:05:09.601773 24541 raft_consensus.cc:2866] T 21c43af5e6844fdaacc8c625fa01fd8d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260925 03:05:09.575979 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 44854us
I20260925 03:05:09.525739 27363 raft_consensus.cc:482] T 8e5819c15eac4544a57e5ee6c9d4f117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:09.519902 27378 raft_consensus.cc:2781] T 48ca6ccd98194cf2af7b5278d15bf4ea P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Leader election won for term 6
I20260925 03:05:09.587692 24511 leader_election.cc:302] T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:05:09.587911 25196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5aec38e66a947c681e6c2a72b05e6ca" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:05:09.589669 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b8e0d260a884ecdb20f79de46c36b19" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:09.584105 27341 raft_consensus.cc:482] T 367277daf8d04778b3fa4ed94d71580d P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:09.570067 27022 raft_consensus.cc:482] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:05:09.592726 24544 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f22ddd7d1c444dc58f45a1fb0b3e02bc"
dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
from {username='slave'} at 127.0.0.1:36786
I20260925 03:05:09.608350 24614 raft_consensus.cc:1261] T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260925 03:05:09.535141 27053 raft_consensus.cc:686] T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c [term 7 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:05:09.500962 27386 consensus_queue.cc:1035] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.093s
I20260925 03:05:09.573045 27343 raft_consensus.cc:482] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:09.593600 27126 consensus_queue.cc:1035] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260925 03:05:09.584046 27385 raft_consensus.cc:3032] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c [term 7 LEADER]: Stepping down as leader of term 7
I20260925 03:05:09.614847 27385 raft_consensus.cc:729] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c [term 7 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:05:09.614988 27402 raft_consensus.cc:2781] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260925 03:05:09.615492 27402 raft_consensus.cc:482] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:05:09.615849 27402 raft_consensus.cc:3037] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Advancing to term 8
I20260925 03:05:09.587755 27267 raft_consensus.cc:504] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:09.621096 27267 leader_election.cc:288] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:09.585089 27190 raft_consensus.cc:2781] T 25102ba08e4e49388f6176badcd10a25 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader election won for term 6
I20260925 03:05:09.621924 27190 raft_consensus.cc:686] T 25102ba08e4e49388f6176badcd10a25 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 LEADER]: Becoming Leader. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:05:09.541775 27273 raft_consensus.cc:504] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:09.597592 27204 raft_consensus.cc:686] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43 [term 7 LEADER]: Becoming Leader. State: Replica: 1bc1f2e62ab44af995165bd5116a9d43, State: Running, Role: LEADER
I20260925 03:05:09.603319 24514 leader_election.cc:302] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:09.517572 27255 raft_consensus.cc:504] T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:09.607596 24511 leader_election.cc:302] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:05:09.606590 24548 raft_consensus.cc:2454] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 6.
W20260925 03:05:09.647070 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 43838.2us
I20260925 03:05:09.647656 24514 leader_election.cc:302] T 11bb46bce3564841b7674e6a693bd2e6 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:05:09.608417 24544 raft_consensus.cc:482] T f22ddd7d1c444dc58f45a1fb0b3e02bc P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 03:05:09.648952 24544 raft_consensus.cc:3037] T f22ddd7d1c444dc58f45a1fb0b3e02bc P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Advancing to term 8
I20260925 03:05:09.613631 24588 leader_election.cc:302] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:05:09.604846 27052 raft_consensus.cc:3037] T a7aa47c48404465b94a36fd819fe3c7a P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Advancing to term 8
I20260925 03:05:09.609824 27022 raft_consensus.cc:3037] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Advancing to term 9
I20260925 03:05:09.609565 27376 raft_consensus.cc:482] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:09.630344 27053 consensus_queue.cc:237] T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:09.613907 27392 raft_consensus.cc:2781] T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260925 03:05:09.634106 26829 leader_election.cc:288] T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:05:09.636225 27259 leader_election.cc:288] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:05:09.623086 27343 raft_consensus.cc:504] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:09.614256 27341 raft_consensus.cc:504] T 367277daf8d04778b3fa4ed94d71580d P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:09.623263 27346 raft_consensus.cc:482] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:09.641373 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b627ecb4a5384ef392aa695e06599c39" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:09.661031 27126 raft_consensus.cc:2726] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260925 03:05:09.658109 27232 raft_consensus.cc:482] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:09.663260 27267 raft_consensus.cc:482] T ceab6b0fccd54e909a64f66fe81a9a1f P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:09.643088 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ac466793d094325ace224708b2158ed" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:09.643378 27204 consensus_queue.cc:237] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 5.8, Last appended by leader: 8, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:09.607458 27363 raft_consensus.cc:504] T 8e5819c15eac4544a57e5ee6c9d4f117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:09.644940 27190 consensus_queue.cc:237] T 25102ba08e4e49388f6176badcd10a25 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:09.692539 26811 raft_consensus.cc:482] T b030d84c670b48d5bff337ccc0c77626 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:09.692736 27369 raft_consensus.cc:3032] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 LEADER]: Stepping down as leader of term 6
I20260925 03:05:09.693112 27369 raft_consensus.cc:729] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:05:09.616109 27385 consensus_queue.cc:260] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:05:09.659765 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 46243.4us
I20260925 03:05:09.693199 26811 raft_consensus.cc:504] T b030d84c670b48d5bff337ccc0c77626 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:09.664779 24514 leader_election.cc:302] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
W20260925 03:05:09.663655 24511 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 56103.3us
I20260925 03:05:09.658583 26842 raft_consensus.cc:482] T 37a157996e944db2906110d87ecb0c54 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:09.676235 25197 raft_consensus.cc:1261] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260925 03:05:09.658691 27377 raft_consensus.cc:482] T 74443a0a1dc34d88a2f42e3b722db405 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:09.658288 27253 raft_consensus.cc:482] T 7993f9c3e2a44f408bbf5d1e395e2306 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:09.652884 27378 raft_consensus.cc:686] T 48ca6ccd98194cf2af7b5278d15bf4ea P eeab93abfdb141a895002ec9f312296c [term 6 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:05:09.678601 27384 leader_election.cc:288] T 25102ba08e4e49388f6176badcd10a25 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 7 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:05:09.660017 27323 raft_consensus.cc:2781] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260925 03:05:09.641897 27273 leader_election.cc:288] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:09.679888 24544 raft_consensus.cc:504] T f22ddd7d1c444dc58f45a1fb0b3e02bc P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:09.682673 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5f9b503b0ab43999d582015a367441e" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:09.683107 24615 raft_consensus.cc:1261] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260925 03:05:09.668744 27392 raft_consensus.cc:482] T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:05:09.671394 27383 raft_consensus.cc:2781] T 11bb46bce3564841b7674e6a693bd2e6 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader election won for term 4
I20260925 03:05:09.690893 27363 leader_election.cc:288] T 8e5819c15eac4544a57e5ee6c9d4f117 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:09.656320 27009 consensus_queue.cc:1035] T 21c43af5e6844fdaacc8c625fa01fd8d P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.030s
I20260925 03:05:09.672583 27403 raft_consensus.cc:2781] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260925 03:05:09.713562 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b627ecb4a5384ef392aa695e06599c39" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:09.738430 24472 raft_consensus.cc:2454] T b627ecb4a5384ef392aa695e06599c39 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 7.
I20260925 03:05:09.717542 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "420163a6cdce412ebc21c25fe3835638" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
W20260925 03:05:09.718706 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 54049.9us
I20260925 03:05:09.718927 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1e85248d3b04b979dd080caeeb02fdb" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:09.719565 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f1fac028a144058b92331c0d9b44ebf" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:09.742914 24545 raft_consensus.cc:2379] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 in current term 8: Already voted for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in this term.
I20260925 03:05:09.743453 24514 leader_election.cc:302] T 1414dcfe53f64a78ae413531eb8ec117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:09.720793 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e5819c15eac4544a57e5ee6c9d4f117" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:09.744481 24547 raft_consensus.cc:2379] T 8e5819c15eac4544a57e5ee6c9d4f117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 in current term 8: Already voted for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in this term.
I20260925 03:05:09.744861 27368 raft_consensus.cc:2726] T 1414dcfe53f64a78ae413531eb8ec117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260925 03:05:09.721874 26946 raft_consensus.cc:482] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:09.745728 27403 raft_consensus.cc:482] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:09.745618 26946 raft_consensus.cc:504] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:09.746292 27403 raft_consensus.cc:3037] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:05:09.727656 27253 raft_consensus.cc:504] T 7993f9c3e2a44f408bbf5d1e395e2306 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:09.747812 26946 leader_election.cc:288] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:09.748844 27253 leader_election.cc:288] T 7993f9c3e2a44f408bbf5d1e395e2306 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:05:09.722455 24511 leader_election.cc:302] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:05:09.750550 26800 raft_consensus.cc:2781] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260925 03:05:09.751014 26800 raft_consensus.cc:482] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:05:09.751363 26800 raft_consensus.cc:3037] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Advancing to term 9
I20260925 03:05:09.751428 24514 leader_election.cc:302] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:09.753072 27368 consensus_queue.cc:1035] T c88cbd39e01f44628a094af516be2833 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260925 03:05:09.724380 24467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c81a1b3b616492ab6c964f45af36077" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:09.715016 27301 raft_consensus.cc:482] T 7993f9c3e2a44f408bbf5d1e395e2306 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:09.755223 27301 raft_consensus.cc:504] T 7993f9c3e2a44f408bbf5d1e395e2306 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:09.725626 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25102ba08e4e49388f6176badcd10a25" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:05:09.728965 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25102ba08e4e49388f6176badcd10a25" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:09.757279 24468 raft_consensus.cc:3037] T 25102ba08e4e49388f6176badcd10a25 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:05:09.735584 24542 raft_consensus.cc:1226] T f22ddd7d1c444dc58f45a1fb0b3e02bc P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 7. Current term is 8. Ops: []
I20260925 03:05:09.758139 27385 raft_consensus.cc:3037] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Advancing to term 8
I20260925 03:05:09.582886 27354 consensus_queue.cc:237] T 43366a0a2f9d4344bf4e37ffd290ef8a P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:09.685652 27346 raft_consensus.cc:504] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:09.762090 27346 leader_election.cc:288] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:09.736524 24618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7691c6b475428c82d6181d693d26f3" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:09.688469 27232 raft_consensus.cc:504] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:09.728354 26842 raft_consensus.cc:504] T 37a157996e944db2906110d87ecb0c54 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:09.763875 27392 raft_consensus.cc:3037] T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Advancing to term 8
I20260925 03:05:09.726436 27279 raft_consensus.cc:2726] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260925 03:05:09.765796 27279 raft_consensus.cc:2726] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260925 03:05:09.735621 27291 raft_consensus.cc:3037] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Advancing to term 8
I20260925 03:05:09.766530 24620 raft_consensus.cc:1261] T 37a157996e944db2906110d87ecb0c54 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260925 03:05:09.766633 24621 raft_consensus.cc:2454] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 9.
I20260925 03:05:09.765945 27378 consensus_queue.cc:237] T 48ca6ccd98194cf2af7b5278d15bf4ea P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:09.764073 24588 leader_election.cc:302] T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:05:09.776294 24514 leader_election.cc:302] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:05:09.803830 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "550118bc1a0641ce9803bdbf43f2069f" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:09.768438 24544 leader_election.cc:288] T f22ddd7d1c444dc58f45a1fb0b3e02bc P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 8 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:09.775769 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7993f9c3e2a44f408bbf5d1e395e2306" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:09.810717 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37a157996e944db2906110d87ecb0c54" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:09.668910 27376 raft_consensus.cc:504] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:09.812176 24615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442d4a29b1d743599f4b6b4871e26668" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:09.819512 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f1fac028a144058b92331c0d9b44ebf" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:09.824419 24613 raft_consensus.cc:2454] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 8.
I20260925 03:05:09.824817 24468 raft_consensus.cc:2454] T 25102ba08e4e49388f6176badcd10a25 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 7.
I20260925 03:05:09.816726 24547 raft_consensus.cc:2454] T 7993f9c3e2a44f408bbf5d1e395e2306 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 5.
I20260925 03:05:09.825690 24615 raft_consensus.cc:2454] T 442d4a29b1d743599f4b6b4871e26668 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 8.
I20260925 03:05:09.828187 24467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7993f9c3e2a44f408bbf5d1e395e2306" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:09.735468 27323 raft_consensus.cc:482] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:05:09.836680 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367277daf8d04778b3fa4ed94d71580d" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
W20260925 03:05:09.833205 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 69216.4us
I20260925 03:05:09.737365 27383 raft_consensus.cc:686] T 11bb46bce3564841b7674e6a693bd2e6 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 LEADER]: Becoming Leader. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:05:09.840830 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5f9b503b0ab43999d582015a367441e" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:09.842237 24616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e5819c15eac4544a57e5ee6c9d4f117" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:09.652386 27364 consensus_queue.cc:237] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:09.810837 26842 leader_election.cc:288] T 37a157996e944db2906110d87ecb0c54 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:05:09.817616 27279 raft_consensus.cc:2781] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Leader election won for term 9
I20260925 03:05:09.750600 27126 consensus_queue.cc:1035] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.091s
I20260925 03:05:09.802979 27249 raft_consensus.cc:482] T 7993f9c3e2a44f408bbf5d1e395e2306 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:09.693950 27369 consensus_queue.cc:260] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.9, Last appended by leader: 9, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:09.893436 24615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35992316cb8d4c028333388eff02ecde" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
W20260925 03:05:09.849841 27322 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 6 current term: 6 is higher.
I20260925 03:05:09.840183 27323 raft_consensus.cc:3037] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Advancing to term 8
W20260925 03:05:09.884146 27405 log.cc:924] Time spent T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.066s user 0.000s sys 0.001s
I20260925 03:05:09.802137 27341 leader_election.cc:288] T 367277daf8d04778b3fa4ed94d71580d P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:09.902345 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b030d84c670b48d5bff337ccc0c77626" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:09.701030 27267 raft_consensus.cc:504] T ceab6b0fccd54e909a64f66fe81a9a1f P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:09.800002 27402 raft_consensus.cc:504] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:09.840855 27343 leader_election.cc:288] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:09.883783 26811 leader_election.cc:288] T b030d84c670b48d5bff337ccc0c77626 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:09.843143 27232 leader_election.cc:288] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:05:09.818431 27252 raft_consensus.cc:504] T 11bb46bce3564841b7674e6a693bd2e6 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:09.903016 24613 raft_consensus.cc:2454] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 9.
I20260925 03:05:09.894285 27304 raft_consensus.cc:482] T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:09.913281 24436 leader_election.cc:302] T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: 1bc1f2e62ab44af995165bd5116a9d43, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:09.824323 27373 consensus_queue.cc:1035] T 5278a60f97da4e10aecd50a8d9a9a43c P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.063s
I20260925 03:05:09.909965 25198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ac466793d094325ace224708b2158ed" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:05:09.907722 24588 leader_election.cc:302] T 25102ba08e4e49388f6176badcd10a25 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5
W20260925 03:05:09.912194 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 106348us
I20260925 03:05:09.727783 27377 raft_consensus.cc:504] T 74443a0a1dc34d88a2f42e3b722db405 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:05:09.887584 24511 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 137335us
W20260925 03:05:09.906755 27268 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.889s user 0.016s sys 0.005s
I20260925 03:05:09.917531 24615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367277daf8d04778b3fa4ed94d71580d" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:09.919171 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c81a1b3b616492ab6c964f45af36077" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:09.906533 24585 leader_election.cc:302] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:05:09.920306 24436 leader_election.cc:302] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:09.929919 24613 raft_consensus.cc:2454] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 6.
I20260925 03:05:09.930513 24436 leader_election.cc:302] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:05:09.931310 24436 leader_election.cc:302] T 442d4a29b1d743599f4b6b4871e26668 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:05:09.886382 27041 raft_consensus.cc:2781] T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260925 03:05:09.941432 27412 consensus_queue.cc:1035] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260925 03:05:09.942224 27273 raft_consensus.cc:2726] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260925 03:05:09.925237 24514 leader_election.cc:302] T 420163a6cdce412ebc21c25fe3835638 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:09.842124 27190 consensus_queue.cc:1035] T c88cbd39e01f44628a094af516be2833 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.072s
I20260925 03:05:09.942911 27273 raft_consensus.cc:2781] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260925 03:05:09.894979 27279 raft_consensus.cc:686] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 LEADER]: Becoming Leader. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:05:09.944015 26946 raft_consensus.cc:2781] T 442d4a29b1d743599f4b6b4871e26668 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260925 03:05:09.937273 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b030d84c670b48d5bff337ccc0c77626" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:09.944818 24546 raft_consensus.cc:2454] T b030d84c670b48d5bff337ccc0c77626 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 8.
I20260925 03:05:09.943326 27126 raft_consensus.cc:482] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 1bc1f2e62ab44af995165bd5116a9d43)
I20260925 03:05:09.946015 24437 leader_election.cc:302] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:05:09.944170 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35992316cb8d4c028333388eff02ecde" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:09.946765 24548 raft_consensus.cc:2454] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 8.
I20260925 03:05:09.947521 27126 raft_consensus.cc:504] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:09.950011 27126 leader_election.cc:288] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
W20260925 03:05:09.950834 27397 log.cc:924] Time spent T 00000000000000000000000000000000 P 2d92cc0732c4490eb7d98d353826cce1: Append to log took a long time: real 0.321s user 0.002s sys 0.000s
I20260925 03:05:09.958423 27077 raft_consensus.cc:2781] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260925 03:05:09.958902 27077 raft_consensus.cc:482] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:05:09.959218 27077 raft_consensus.cc:3037] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Advancing to term 10
I20260925 03:05:09.962628 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5f9b503b0ab43999d582015a367441e" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:09.963244 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37a157996e944db2906110d87ecb0c54" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:09.963354 24466 raft_consensus.cc:2454] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 9.
I20260925 03:05:09.968374 27252 leader_election.cc:288] T 11bb46bce3564841b7674e6a693bd2e6 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:05:09.921581 27346 consensus_queue.cc:1035] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.049s
I20260925 03:05:09.925966 27304 raft_consensus.cc:504] T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:09.973623 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b030d84c670b48d5bff337ccc0c77626" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:09.974004 27304 leader_election.cc:288] T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:09.941465 27041 raft_consensus.cc:482] T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:05:09.975524 27041 raft_consensus.cc:3037] T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Advancing to term 8
I20260925 03:05:09.977101 26900 raft_consensus.cc:3037] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Advancing to term 9
I20260925 03:05:09.886641 27386 consensus_queue.cc:1035] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.046s
I20260925 03:05:09.943383 27273 raft_consensus.cc:482] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:09.948088 27384 raft_consensus.cc:2781] T 25102ba08e4e49388f6176badcd10a25 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Leader election won for term 7
W20260925 03:05:09.985656 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 60605.7us
I20260925 03:05:09.918349 27369 raft_consensus.cc:3037] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:05:09.990054 24437 leader_election.cc:302] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:05:10.002360 24473 raft_consensus.cc:1261] T ceab6b0fccd54e909a64f66fe81a9a1f P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 4. (term mismatch)
I20260925 03:05:10.002760 24473 pending_rounds.cc:76] T ceab6b0fccd54e909a64f66fe81a9a1f P 869b0a8575b54d08ae2817fad5dd42d5: Aborting all ops after (but not including) 3
W20260925 03:05:09.990589 24585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.4:43979, user_credentials={real_user=slave}} blocked reactor thread for 84165.7us
I20260925 03:05:10.003379 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bb46bce3564841b7674e6a693bd2e6" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:10.003587 24585 leader_election.cc:302] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:05:10.003983 24471 raft_consensus.cc:3037] T 11bb46bce3564841b7674e6a693bd2e6 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:05:09.992451 27190 raft_consensus.cc:2726] T 420163a6cdce412ebc21c25fe3835638 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260925 03:05:10.005077 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "550118bc1a0641ce9803bdbf43f2069f" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:10.006366 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f22ddd7d1c444dc58f45a1fb0b3e02bc" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: true dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:10.006982 24466 raft_consensus.cc:3037] T f22ddd7d1c444dc58f45a1fb0b3e02bc P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Advancing to term 8
I20260925 03:05:09.944448 26946 raft_consensus.cc:482] T 442d4a29b1d743599f4b6b4871e26668 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:10.007750 26946 raft_consensus.cc:3037] T 442d4a29b1d743599f4b6b4871e26668 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Advancing to term 8
I20260925 03:05:10.009153 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06f4a7eaf6534e4986c9fd35911038e4" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:10.010207 27376 leader_election.cc:288] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:05:10.011447 24472 raft_consensus.cc:1261] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260925 03:05:09.991389 24620 raft_consensus.cc:2454] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 9.
I20260925 03:05:09.922487 27363 raft_consensus.cc:2726] T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260925 03:05:09.985299 24588 leader_election.cc:302] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:05:10.002960 24473 pending_rounds.cc:97] T ceab6b0fccd54e909a64f66fe81a9a1f P 869b0a8575b54d08ae2817fad5dd42d5: Aborting uncommitted NO_OP operation due to leader change: 7.4
I20260925 03:05:10.033010 24473 raft_consensus.cc:2866] T ceab6b0fccd54e909a64f66fe81a9a1f P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260925 03:05:09.895771 27249 raft_consensus.cc:504] T 7993f9c3e2a44f408bbf5d1e395e2306 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:10.015021 24468 raft_consensus.cc:1261] T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260925 03:05:09.995190 25198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71f461a6e2ff434993b78a2d077f0abd" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:05:10.006218 27272 raft_consensus.cc:2781] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260925 03:05:09.941249 27268 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.924s user 0.016s sys 0.005s
I20260925 03:05:10.010455 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b030d84c670b48d5bff337ccc0c77626" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:10.016990 23300 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260925 03:05:10.046597 24615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f22ddd7d1c444dc58f45a1fb0b3e02bc" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: true dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:05:10.047233 24615 raft_consensus.cc:3032] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c [term 7 LEADER]: Stepping down as leader of term 7
I20260925 03:05:10.047531 24615 raft_consensus.cc:729] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c [term 7 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
W20260925 03:05:10.051102 24511 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.4:43979, user_credentials={real_user=slave}} blocked reactor thread for 61395.4us
I20260925 03:05:10.016497 24472 raft_consensus.cc:1226] T 293ef69caf514ac98510a68495f54eda P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 6. Current term is 7. Ops: [6.9-6.9]
I20260925 03:05:10.052387 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74443a0a1dc34d88a2f42e3b722db405" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:10.035914 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bb46bce3564841b7674e6a693bd2e6" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:05:10.053149 24585 leader_election.cc:302] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 1bc1f2e62ab44af995165bd5116a9d43, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:05:09.995837 27273 raft_consensus.cc:3037] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Advancing to term 8
I20260925 03:05:09.994457 26756 raft_consensus.cc:2726] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260925 03:05:10.077317 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e725b82985746b39c0cc8f671c95353" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:10.062716 24541 raft_consensus.cc:1261] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20260925 03:05:10.076432 24471 raft_consensus.cc:2454] T 11bb46bce3564841b7674e6a693bd2e6 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 5.
I20260925 03:05:10.083034 24473 raft_consensus.cc:1261] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260925 03:05:10.087815 24467 raft_consensus.cc:2454] T 7993f9c3e2a44f408bbf5d1e395e2306 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 6.
I20260925 03:05:10.040373 27243 raft_consensus.cc:2726] T c5aec38e66a947c681e6c2a72b05e6ca P eeab93abfdb141a895002ec9f312296c [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260925 03:05:10.040844 27346 raft_consensus.cc:482] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260925 03:05:10.091799 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 106575us
I20260925 03:05:10.107275 24588 leader_election.cc:302] T 37a157996e944db2906110d87ecb0c54 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:05:10.095399 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "293ef69caf514ac98510a68495f54eda" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:05:10.078037 24615 consensus_queue.cc:260] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:10.109163 24615 raft_consensus.cc:3037] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Advancing to term 8
I20260925 03:05:10.097295 24514 leader_election.cc:302] T 550118bc1a0641ce9803bdbf43f2069f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:10.096560 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367277daf8d04778b3fa4ed94d71580d" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:10.045831 27272 raft_consensus.cc:482] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260925 03:05:10.106755 24585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.4:43979, user_credentials={real_user=slave}} blocked reactor thread for 53711.3us
W20260925 03:05:10.091595 27347 log.cc:924] Time spent T 21c43af5e6844fdaacc8c625fa01fd8d P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.073s user 0.001s sys 0.000s
I20260925 03:05:10.137130 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7993f9c3e2a44f408bbf5d1e395e2306" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:09.944476 27279 consensus_queue.cc:237] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:10.082839 27384 raft_consensus.cc:686] T 25102ba08e4e49388f6176badcd10a25 P eeab93abfdb141a895002ec9f312296c [term 7 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:05:09.993266 27383 consensus_queue.cc:237] T 11bb46bce3564841b7674e6a693bd2e6 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:10.073221 27343 consensus_queue.cc:1035] T e6b6ec010caf4689b8c90449764f5cf9 P 1bc1f2e62ab44af995165bd5116a9d43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.069s
I20260925 03:05:10.140549 27232 raft_consensus.cc:2726] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260925 03:05:10.127759 27346 raft_consensus.cc:504] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:10.169569 27384 consensus_queue.cc:237] T 25102ba08e4e49388f6176badcd10a25 P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:10.139290 26970 raft_consensus.cc:2781] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260925 03:05:10.127208 27272 raft_consensus.cc:3037] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Advancing to term 9
W20260925 03:05:10.506006 27291 consensus_meta.cc:287] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c: Time spent flushing consensus metadata: real 0.739s user 0.007s sys 0.002s
I20260925 03:05:10.145160 27368 raft_consensus.cc:2726] T 550118bc1a0641ce9803bdbf43f2069f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260925 03:05:10.504449 27041 consensus_meta.cc:287] T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c: Time spent flushing consensus metadata: real 0.528s user 0.007s sys 0.000s
I20260925 03:05:10.435883 24466 raft_consensus.cc:2454] T f22ddd7d1c444dc58f45a1fb0b3e02bc P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 8.
I20260925 03:05:10.215835 24615 raft_consensus.cc:2454] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 8.
W20260925 03:05:10.551666 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 444440us
W20260925 03:05:10.550613 27077 consensus_meta.cc:287] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5: Time spent flushing consensus metadata: real 0.591s user 0.008s sys 0.004s
I20260925 03:05:10.553125 27328 consensus_queue.cc:1035] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260925 03:05:10.553529 27077 raft_consensus.cc:504] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:10.554406 24588 leader_election.cc:302] T 11bb46bce3564841b7674e6a693bd2e6 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:05:10.555229 27328 raft_consensus.cc:2781] T 11bb46bce3564841b7674e6a693bd2e6 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Leader election won for term 5
I20260925 03:05:10.555716 27328 raft_consensus.cc:686] T 11bb46bce3564841b7674e6a693bd2e6 P eeab93abfdb141a895002ec9f312296c [term 5 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:05:10.556559 27328 consensus_queue.cc:237] T 11bb46bce3564841b7674e6a693bd2e6 P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:10.558492 25162 leader_election.cc:302] T 367277daf8d04778b3fa4ed94d71580d P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:10.559356 27399 raft_consensus.cc:2726] T 367277daf8d04778b3fa4ed94d71580d P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260925 03:05:10.560390 25162 leader_election.cc:302] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:05:10.561231 24469 raft_consensus.cc:2354] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 9. Current term is 10.
I20260925 03:05:10.561661 27399 raft_consensus.cc:2781] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260925 03:05:10.562045 27399 raft_consensus.cc:482] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:05:10.562353 27399 raft_consensus.cc:3037] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:05:10.563301 27077 leader_election.cc:288] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 10 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:10.565733 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceab6b0fccd54e909a64f66fe81a9a1f" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:10.566470 24548 raft_consensus.cc:2454] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 8.
W20260925 03:05:10.567433 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 436031us
I20260925 03:05:10.151419 27301 leader_election.cc:288] T 7993f9c3e2a44f408bbf5d1e395e2306 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
W20260925 03:05:10.529686 27323 consensus_meta.cc:287] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P eeab93abfdb141a895002ec9f312296c: Time spent flushing consensus metadata: real 0.615s user 0.009s sys 0.000s
W20260925 03:05:10.495513 27419 log.cc:924] Time spent T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.203s user 0.000s sys 0.000s
W20260925 03:05:10.499466 27052 consensus_meta.cc:287] T a7aa47c48404465b94a36fd819fe3c7a P eeab93abfdb141a895002ec9f312296c: Time spent flushing consensus metadata: real 0.828s user 0.009s sys 0.000s
W20260925 03:05:10.503401 27385 consensus_meta.cc:287] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c: Time spent flushing consensus metadata: real 0.745s user 0.006s sys 0.001s
W20260925 03:05:10.454232 27424 log.cc:924] Time spent T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.177s user 0.001s sys 0.000s
W20260925 03:05:10.469461 27410 log.cc:924] Time spent T 25102ba08e4e49388f6176badcd10a25 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.538s user 0.001s sys 0.000s
W20260925 03:05:10.170857 27411 log.cc:924] Time spent T 21c43af5e6844fdaacc8c625fa01fd8d P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.059s user 0.000s sys 0.000s
I20260925 03:05:10.604447 27291 consensus_queue.cc:1035] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
W20260925 03:05:10.605336 26900 consensus_meta.cc:287] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c: Time spent flushing consensus metadata: real 0.628s user 0.009s sys 0.000s
W20260925 03:05:10.605883 27022 consensus_meta.cc:287] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c: Time spent flushing consensus metadata: real 0.932s user 0.010s sys 0.000s
I20260925 03:05:10.606253 27022 raft_consensus.cc:504] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:10.608264 27022 leader_election.cc:288] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 9 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:05:10.602021 24616 raft_consensus.cc:2454] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 8.
I20260925 03:05:10.572652 27368 consensus_queue.cc:1035] T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20260925 03:05:10.614740 27392 consensus_meta.cc:287] T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent flushing consensus metadata: real 0.850s user 0.010s sys 0.001s
I20260925 03:05:10.615211 27392 raft_consensus.cc:504] T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:05:10.618182 27390 log.cc:924] Time spent T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 1.310s user 0.003s sys 0.006s
I20260925 03:05:10.572660 27386 consensus_queue.cc:1035] T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260925 03:05:10.379899 27304 raft_consensus.cc:482] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:10.618904 25199 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49794 (request call id 295) took 1047 ms. Trace:
I20260925 03:05:10.575321 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b030d84c670b48d5bff337ccc0c77626" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:05:10.619656 24546 raft_consensus.cc:3037] T b030d84c670b48d5bff337ccc0c77626 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Advancing to term 10
I20260925 03:05:10.573900 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7993f9c3e2a44f408bbf5d1e395e2306" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:10.619313 25199 rpcz_store.cc:276] 0925 03:05:09.571412 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:05:09.571546 (+ 134us) service_pool.cc:227] Handling call
0925 03:05:09.588614 (+ 17068us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:05:09.697000 (+108386us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:05:09.697033 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:05:09.697073 (+ 40us) raft_consensus.cc:1631] Marking committed up to 6
0925 03:05:09.697123 (+ 50us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:05:10.618671 (+921548us) spinlock_profiling.cc:243] Waited 108 ms on lock 0x7b54005bf754. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d0f51 00007f43764cf99e 00007f437b4112e9 00007f4374af7684 00007f4374af7612 00007f4374af7541 00007f4374af74bc 00007f4374af6772 00007f43748fa94c 00007f43748f9d86
0925 03:05:10.618716 (+ 45us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:05:10.618842 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":248829824}
I20260925 03:05:10.575317 27346 leader_election.cc:288] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:10.576504 24437 leader_election.cc:302] T ceab6b0fccd54e909a64f66fe81a9a1f P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
W20260925 03:05:10.626207 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 49784.5us
I20260925 03:05:10.161530 27403 raft_consensus.cc:504] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:10.504248 27249 leader_election.cc:288] T 7993f9c3e2a44f408bbf5d1e395e2306 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:10.570364 27041 raft_consensus.cc:504] T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:10.542332 27363 raft_consensus.cc:482] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:10.166747 26842 raft_consensus.cc:2726] T 37a157996e944db2906110d87ecb0c54 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260925 03:05:10.195757 26800 raft_consensus.cc:504] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:10.584545 27052 raft_consensus.cc:504] T a7aa47c48404465b94a36fd819fe3c7a P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:10.276018 27402 leader_election.cc:288] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 8 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:05:10.537096 26970 raft_consensus.cc:482] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:05:10.571312 27412 consensus_queue.cc:1035] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.017s
I20260925 03:05:10.580789 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5278a60f97da4e10aecd50a8d9a9a43c" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:10.578702 27323 raft_consensus.cc:504] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:10.642868 26800 leader_election.cc:288] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 9 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:10.643831 24544 raft_consensus.cc:1226] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 8. Current term is 9. Ops: []
I20260925 03:05:10.644142 27323 leader_election.cc:288] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 8 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:05:10.591459 27255 leader_election.cc:288] T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:05:10.640825 26970 raft_consensus.cc:3037] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Advancing to term 10
I20260925 03:05:10.644827 27363 raft_consensus.cc:504] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:10.645632 27052 leader_election.cc:288] T a7aa47c48404465b94a36fd819fe3c7a P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 8 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:05:10.639735 27041 leader_election.cc:288] T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 8 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:05:10.641024 24618 raft_consensus.cc:1226] T a7aa47c48404465b94a36fd819fe3c7a P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 7. Current term is 8. Ops: []
W20260925 03:05:10.592139 27273 consensus_meta.cc:287] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5: Time spent flushing consensus metadata: real 0.501s user 0.005s sys 0.001s
I20260925 03:05:10.649309 27273 raft_consensus.cc:504] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:10.651368 24620 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "04ac285175464fbda67211d9574db415"
dest_uuid: "eeab93abfdb141a895002ec9f312296c"
mode: GRACEFUL
new_leader_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
from {username='slave'} at 127.0.0.1:44570
I20260925 03:05:10.651914 27273 leader_election.cc:288] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 8 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:10.652032 24620 raft_consensus.cc:595] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [term 7 LEADER]: Received request to transfer leadership to TS a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:05:10.654266 24617 raft_consensus.cc:1226] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 7. Current term is 8. Ops: []
W20260925 03:05:10.635133 24436 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.649s user 0.003s sys 0.021s
W20260925 03:05:10.656438 24436 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.725s user 0.003s sys 0.024s
W20260925 03:05:10.659021 24436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 727736us
I20260925 03:05:10.591944 24467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e725b82985746b39c0cc8f671c95353" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:10.530159 27377 leader_election.cc:288] T 74443a0a1dc34d88a2f42e3b722db405 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:05:10.444875 27267 leader_election.cc:288] T ceab6b0fccd54e909a64f66fe81a9a1f P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:10.598562 24511 leader_election.cc:302] T b030d84c670b48d5bff337ccc0c77626 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
W20260925 03:05:10.665776 24511 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 67317.6us
I20260925 03:05:10.667218 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60d45b7d467d4e1ab92367f6d7d52b15" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:10.667739 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f1fac028a144058b92331c0d9b44ebf" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:05:10.668474 24543 raft_consensus.cc:2379] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 in current term 8: Already voted for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in this term.
I20260925 03:05:10.619131 27304 raft_consensus.cc:504] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:10.567637 26829 raft_consensus.cc:482] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260925 03:05:10.600289 24472 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.526s user 0.004s sys 0.011s
W20260925 03:05:10.673755 24472 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.600s user 0.004s sys 0.011s
I20260925 03:05:10.608880 24613 raft_consensus.cc:1226] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 8. Current term is 9. Ops: []
I20260925 03:05:10.502390 26946 raft_consensus.cc:504] T 442d4a29b1d743599f4b6b4871e26668 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:10.624178 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74443a0a1dc34d88a2f42e3b722db405" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:10.624855 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b8e0d260a884ecdb20f79de46c36b19" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:05:10.628461 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b8e0d260a884ecdb20f79de46c36b19" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:10.631645 27291 raft_consensus.cc:482] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:10.622357 27272 raft_consensus.cc:504] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:10.723750 27267 raft_consensus.cc:482] T 550118bc1a0641ce9803bdbf43f2069f P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:10.662405 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74443a0a1dc34d88a2f42e3b722db405" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:10.726451 27427 raft_consensus.cc:2781] T ceab6b0fccd54e909a64f66fe81a9a1f P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260925 03:05:10.726552 24620 raft_consensus.cc:2454] T 74443a0a1dc34d88a2f42e3b722db405 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 6.
I20260925 03:05:10.726934 27427 raft_consensus.cc:482] T ceab6b0fccd54e909a64f66fe81a9a1f P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:05:10.727262 27427 raft_consensus.cc:3037] T ceab6b0fccd54e909a64f66fe81a9a1f P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Advancing to term 9
I20260925 03:05:10.727649 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b030d84c670b48d5bff337ccc0c77626" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:05:10.728192 24620 raft_consensus.cc:3037] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Advancing to term 10
I20260925 03:05:10.637506 24546 raft_consensus.cc:2454] T b030d84c670b48d5bff337ccc0c77626 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 10.
I20260925 03:05:10.670480 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74443a0a1dc34d88a2f42e3b722db405" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:10.599440 25196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e068a0f80941dc9138cee063f53bab" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:05:10.722853 24511 leader_election.cc:302] T f22ddd7d1c444dc58f45a1fb0b3e02bc P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:05:10.647058 27363 leader_election.cc:288] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:10.723943 27273 raft_consensus.cc:482] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:10.721778 24540 raft_consensus.cc:3037] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Advancing to term 8
I20260925 03:05:10.722623 24468 raft_consensus.cc:2454] T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 6.
I20260925 03:05:10.725752 27267 raft_consensus.cc:504] T 550118bc1a0641ce9803bdbf43f2069f P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:10.760375 24437 leader_election.cc:302] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:05:10.726208 27272 leader_election.cc:288] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 9 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:10.738118 24436 leader_election.cc:302] T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:05:10.732861 27255 raft_consensus.cc:2741] T b030d84c670b48d5bff337ccc0c77626 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 10 FOLLOWER]: Leader pre-election decision vote started in defunct term 8: won
I20260925 03:05:10.750926 24467 raft_consensus.cc:2454] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 7.
I20260925 03:05:10.726193 26946 leader_election.cc:288] T 442d4a29b1d743599f4b6b4871e26668 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 8 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:10.754676 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c23337a7cb74184a0e57b7095cb6a11" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:05:10.722199 24466 raft_consensus.cc:3037] T 7b8e0d260a884ecdb20f79de46c36b19 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Advancing to term 8
I20260925 03:05:10.763537 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35992316cb8d4c028333388eff02ecde" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:05:10.748768 24620 raft_consensus.cc:2454] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 10.
I20260925 03:05:10.737030 27291 raft_consensus.cc:504] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:10.770660 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442d4a29b1d743599f4b6b4871e26668" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:05:10.781474 24548 raft_consensus.cc:2379] T 442d4a29b1d743599f4b6b4871e26668 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 in current term 8: Already voted for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in this term.
I20260925 03:05:10.771579 24543 raft_consensus.cc:3037] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Advancing to term 9
I20260925 03:05:10.772732 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceab6b0fccd54e909a64f66fe81a9a1f" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:10.773335 27432 consensus_queue.cc:1035] T 37a157996e944db2906110d87ecb0c54 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.029s
I20260925 03:05:10.788618 24464 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1402) took 1263 ms. Trace:
I20260925 03:05:10.789494 24464 rpcz_store.cc:276] 0925 03:05:09.525155 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:05:09.526412 (+ 1257us) service_pool.cc:227] Handling call
0925 03:05:09.526701 (+ 289us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:05:09.695293 (+168592us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:05:09.695321 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:05:09.695358 (+ 37us) raft_consensus.cc:1631] Marking committed up to 4
0925 03:05:09.695390 (+ 32us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:05:10.788369 (+1092979us) spinlock_profiling.cc:243] Waited 168 ms on lock 0x7b54003ec1d4. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d0f51 00007f43764cf99e 00007f437b4112e9 00007f4374af7684 00007f4374af7612 00007f4374af7541 00007f4374af74bc 00007f4374af6772 00007f43748fa94c 00007f43748f9d86
0925 03:05:10.788420 (+ 51us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:05:10.788553 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":387305216}
I20260925 03:05:10.791056 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abd5b8dc1b7e49e3aa9af900d73ff0e7" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:10.791662 24464 raft_consensus.cc:3037] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Advancing to term 8
I20260925 03:05:10.764318 27022 raft_consensus.cc:482] T c88cbd39e01f44628a094af516be2833 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:10.802799 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7aa47c48404465b94a36fd819fe3c7a" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:10.769381 27368 raft_consensus.cc:2781] T f22ddd7d1c444dc58f45a1fb0b3e02bc P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Leader election won for term 8
I20260925 03:05:10.771664 24588 leader_election.cc:302] T 7993f9c3e2a44f408bbf5d1e395e2306 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:05:10.793967 24540 raft_consensus.cc:2454] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 8.
I20260925 03:05:10.800438 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c23337a7cb74184a0e57b7095cb6a11" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:10.806291 24471 raft_consensus.cc:3037] T a7aa47c48404465b94a36fd819fe3c7a P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Advancing to term 8
I20260925 03:05:10.811517 24473 raft_consensus.cc:3037] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Advancing to term 9
I20260925 03:05:10.736474 26829 raft_consensus.cc:504] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:10.779582 27363 raft_consensus.cc:2781] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [term 10 FOLLOWER]: Leader election won for term 10
I20260925 03:05:10.801296 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f1fac028a144058b92331c0d9b44ebf" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:05:10.774214 27392 leader_election.cc:288] T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 8 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:10.809290 24466 raft_consensus.cc:2454] T 7b8e0d260a884ecdb20f79de46c36b19 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 8.
I20260925 03:05:10.804612 24464 raft_consensus.cc:2454] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 8.
I20260925 03:05:10.812143 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b627ecb4a5384ef392aa695e06599c39" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:05:10.803791 24539 raft_consensus.cc:1261] T 48ca6ccd98194cf2af7b5278d15bf4ea P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260925 03:05:10.860930 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7aa47c48404465b94a36fd819fe3c7a" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:05:10.861965 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e725b82985746b39c0cc8f671c95353" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:10.816854 24543 raft_consensus.cc:2454] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 9.
I20260925 03:05:10.796007 27077 raft_consensus.cc:2781] T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260925 03:05:10.824594 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abd5b8dc1b7e49e3aa9af900d73ff0e7" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:05:10.833170 27384 raft_consensus.cc:2781] T 7993f9c3e2a44f408bbf5d1e395e2306 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260925 03:05:10.848713 24467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b627ecb4a5384ef392aa695e06599c39" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:10.852056 27304 leader_election.cc:288] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:10.869850 24437 leader_election.cc:302] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:05:10.870223 24548 raft_consensus.cc:1261] T 43366a0a2f9d4344bf4e37ffd290ef8a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260925 03:05:10.871085 24543 raft_consensus.cc:1261] T 0be4e0aa984f4f2b81a58e382e434ad5 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260925 03:05:10.866547 24539 raft_consensus.cc:1261] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20260925 03:05:10.867034 24473 raft_consensus.cc:2454] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 9.
I20260925 03:05:10.875258 24473 raft_consensus.cc:1261] T 43366a0a2f9d4344bf4e37ffd290ef8a P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260925 03:05:10.876993 24464 raft_consensus.cc:1226] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 8. Current term is 9. Ops: []
I20260925 03:05:10.868655 24620 raft_consensus.cc:2454] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 8.
I20260925 03:05:10.878844 24436 leader_election.cc:302] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:05:10.869238 27384 raft_consensus.cc:482] T 7993f9c3e2a44f408bbf5d1e395e2306 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:05:10.755416 27399 raft_consensus.cc:504] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:10.887328 27384 raft_consensus.cc:3037] T 7993f9c3e2a44f408bbf5d1e395e2306 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Advancing to term 7
W20260925 03:05:10.869848 27416 log.cc:924] Time spent T 21c43af5e6844fdaacc8c625fa01fd8d P 869b0a8575b54d08ae2817fad5dd42d5: Append to log took a long time: real 0.083s user 0.000s sys 0.000s
I20260925 03:05:10.888296 27346 raft_consensus.cc:2781] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Leader election won for term 9
I20260925 03:05:10.889016 27346 raft_consensus.cc:686] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 LEADER]: Becoming Leader. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:05:10.889813 27346 consensus_queue.cc:237] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:10.892416 26946 raft_consensus.cc:2781] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Leader election won for term 8
I20260925 03:05:10.892995 26946 raft_consensus.cc:686] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 LEADER]: Becoming Leader. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:05:10.765607 27273 raft_consensus.cc:504] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:10.892545 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35992316cb8d4c028333388eff02ecde" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:05:10.899012 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60d45b7d467d4e1ab92367f6d7d52b15" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:10.899713 24620 raft_consensus.cc:2454] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 9.
I20260925 03:05:10.892554 24467 raft_consensus.cc:1261] T 293ef69caf514ac98510a68495f54eda P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Refusing update from remote peer 1bc1f2e62ab44af995165bd5116a9d43: Log matching property violated. Preceding OpId in replica: term: 5 index: 8. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20260925 03:05:10.917309 24471 raft_consensus.cc:2454] T a7aa47c48404465b94a36fd819fe3c7a P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 8.
W20260925 03:05:10.925866 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 70142.5us
I20260925 03:05:10.928014 24588 leader_election.cc:302] T 74443a0a1dc34d88a2f42e3b722db405 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:05:10.928073 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06f4a7eaf6534e4986c9fd35911038e4" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:05:10.929711 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7993f9c3e2a44f408bbf5d1e395e2306" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
W20260925 03:05:10.932032 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 98973.3us
I20260925 03:05:10.933116 24586 leader_election.cc:302] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:05:10.869669 27077 raft_consensus.cc:482] T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:05:10.769276 27431 consensus_queue.cc:1035] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.145s
I20260925 03:05:10.936550 27291 leader_election.cc:288] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
W20260925 03:05:10.937065 27369 consensus_meta.cc:287] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent flushing consensus metadata: real 0.899s user 0.010s sys 0.000s
I20260925 03:05:10.937968 24545 raft_consensus.cc:2396] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 1bc1f2e62ab44af995165bd5116a9d43 for term 7 because replica has last-logged OpId of term: 6 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 8.
I20260925 03:05:10.941562 24615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442d4a29b1d743599f4b6b4871e26668" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:05:10.813462 27022 raft_consensus.cc:504] T c88cbd39e01f44628a094af516be2833 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:10.944386 27022 leader_election.cc:288] T c88cbd39e01f44628a094af516be2833 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:05:10.879027 27368 raft_consensus.cc:686] T f22ddd7d1c444dc58f45a1fb0b3e02bc P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 LEADER]: Becoming Leader. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:05:10.929039 27363 raft_consensus.cc:686] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [term 10 LEADER]: Becoming Leader. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:05:10.950013 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7993f9c3e2a44f408bbf5d1e395e2306" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:10.951351 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e068a0f80941dc9138cee063f53bab" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:10.952463 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71f461a6e2ff434993b78a2d077f0abd" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:10.945533 27368 consensus_queue.cc:237] T f22ddd7d1c444dc58f45a1fb0b3e02bc P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:10.959803 27384 raft_consensus.cc:504] T 7993f9c3e2a44f408bbf5d1e395e2306 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:10.893774 26946 consensus_queue.cc:237] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:05:10.970491 25163 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.22.193.2:36389 timed out after 1.706s (SENT) [suppressed 6 similar messages]
W20260925 03:05:10.973249 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 45328.5us
I20260925 03:05:10.974740 24588 leader_election.cc:302] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:05:10.976754 24468 raft_consensus.cc:1261] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 4. (term mismatch)
I20260925 03:05:10.977159 24468 pending_rounds.cc:76] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5: Aborting all ops after (but not including) 3
I20260925 03:05:10.977432 24468 pending_rounds.cc:97] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5: Aborting uncommitted NO_OP operation due to leader change: 7.4
I20260925 03:05:10.977654 24468 raft_consensus.cc:2866] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260925 03:05:10.977090 24615 raft_consensus.cc:2454] T 442d4a29b1d743599f4b6b4871e26668 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 8.
I20260925 03:05:10.979455 24588 leader_election.cc:302] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:05:10.979316 24615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5278a60f97da4e10aecd50a8d9a9a43c" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:10.989265 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e725b82985746b39c0cc8f671c95353" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:10.989748 24613 raft_consensus.cc:2379] T 7993f9c3e2a44f408bbf5d1e395e2306 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 in current term 7: Already voted for candidate eeab93abfdb141a895002ec9f312296c in this term.
I20260925 03:05:10.990108 24615 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "25102ba08e4e49388f6176badcd10a25"
dest_uuid: "eeab93abfdb141a895002ec9f312296c"
mode: GRACEFUL
new_leader_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
from {username='slave'} at 127.0.0.1:44570
I20260925 03:05:10.990882 24615 raft_consensus.cc:595] T 25102ba08e4e49388f6176badcd10a25 P eeab93abfdb141a895002ec9f312296c [term 7 LEADER]: Received request to transfer leadership to TS a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:05:10.949385 27077 raft_consensus.cc:3037] T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Advancing to term 7
W20260925 03:05:10.993139 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 422628us
W20260925 03:05:10.993300 24436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 114548us
W20260925 03:05:10.995334 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 62297.4us
I20260925 03:05:10.983060 27403 leader_election.cc:288] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 7 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:10.986246 27325 raft_consensus.cc:2781] T 74443a0a1dc34d88a2f42e3b722db405 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260925 03:05:11.004019 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06f4a7eaf6534e4986c9fd35911038e4" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:11.010999 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b8e0d260a884ecdb20f79de46c36b19" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:11.011605 24614 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38564 (request call id 1474) took 1488 ms. Trace:
I20260925 03:05:11.011763 24464 raft_consensus.cc:2354] T 7b8e0d260a884ecdb20f79de46c36b19 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 7. Current term is 8.
I20260925 03:05:11.005334 24615 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "442d4a29b1d743599f4b6b4871e26668"
dest_uuid: "eeab93abfdb141a895002ec9f312296c"
mode: GRACEFUL
new_leader_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
from {username='slave'} at 127.0.0.1:44570
I20260925 03:05:11.005679 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e8af2641e124f9589c5ccfdac785f63" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:11.009511 24471 raft_consensus.cc:1226] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 7. Current term is 8. Ops: []
I20260925 03:05:11.012022 24614 rpcz_store.cc:276] 0925 03:05:09.523248 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:05:09.523419 (+ 171us) service_pool.cc:227] Handling call
0925 03:05:09.561083 (+ 37664us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:05:09.608628 (+ 47545us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:05:11.011335 (+1402707us) spinlock_profiling.cc:243] Waited 46.9 ms on lock 0x7b54000735d4. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d0f51 00007f43764cf99e 00007f437b4112e9 00007f4374af7684 00007f4374af7612 00007f4374af7541 00007f4374af74bc 00007f4374af6772 00007f43748fa94c 00007f43748f9d86
0925 03:05:11.011538 (+ 203us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":107954560}
I20260925 03:05:11.013804 24473 raft_consensus.cc:3037] T 9e8af2641e124f9589c5ccfdac785f63 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Advancing to term 8
I20260925 03:05:11.015055 24615 raft_consensus.cc:595] T 442d4a29b1d743599f4b6b4871e26668 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Received request to transfer leadership to TS a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:05:11.015379 24615 raft_consensus.cc:603] T 442d4a29b1d743599f4b6b4871e26668 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260925 03:05:11.015128 24588 leader_election.cc:302] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:05:10.990478 27291 raft_consensus.cc:2781] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260925 03:05:11.019124 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "550118bc1a0641ce9803bdbf43f2069f" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:11.023293 24473 raft_consensus.cc:2454] T 9e8af2641e124f9589c5ccfdac785f63 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 8.
I20260925 03:05:11.020207 24586 leader_election.cc:302] T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:05:10.945828 27363 consensus_queue.cc:237] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:11.009140 27379 raft_consensus.cc:2781] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Leader election won for term 8
I20260925 03:05:11.009001 27262 raft_consensus.cc:2781] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Leader election won for term 8
I20260925 03:05:11.019368 27325 raft_consensus.cc:482] T 74443a0a1dc34d88a2f42e3b722db405 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:05:11.017676 27291 raft_consensus.cc:482] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Starting leader election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:05:11.067152 26756 raft_consensus.cc:2781] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Leader election won for term 9
I20260925 03:05:11.071877 27325 raft_consensus.cc:3037] T 74443a0a1dc34d88a2f42e3b722db405 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:05:11.081597 27401 raft_consensus.cc:979] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c: : Instructing follower a04a37fc4aa7460cbfdbeeb4e451e1f5 to start an election
I20260925 03:05:11.025476 27190 raft_consensus.cc:482] T b4ebd2b043024a548005c8b6b00b0449 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1bc1f2e62ab44af995165bd5116a9d43)
I20260925 03:05:11.082154 27401 raft_consensus.cc:1067] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [term 7 LEADER]: Signalling peer a04a37fc4aa7460cbfdbeeb4e451e1f5 to start an election
I20260925 03:05:11.082335 27190 raft_consensus.cc:504] T b4ebd2b043024a548005c8b6b00b0449 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:11.084837 27262 raft_consensus.cc:686] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [term 8 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:05:11.080155 27291 raft_consensus.cc:3037] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Advancing to term 8
I20260925 03:05:11.085796 27262 consensus_queue.cc:237] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:11.089303 27190 leader_election.cc:288] T b4ebd2b043024a548005c8b6b00b0449 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:05:11.054834 27041 raft_consensus.cc:2726] T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260925 03:05:11.008845 27364 consensus_queue.cc:1035] T 48ca6ccd98194cf2af7b5278d15bf4ea P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.057s
I20260925 03:05:11.072906 27379 raft_consensus.cc:686] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P eeab93abfdb141a895002ec9f312296c [term 8 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:05:11.088531 27267 leader_election.cc:288] T 550118bc1a0641ce9803bdbf43f2069f P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:11.056640 27053 consensus_queue.cc:1035] T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.060s
I20260925 03:05:11.132939 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9c33fd100ff44c1a3247319a324a044" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:11.133745 24541 raft_consensus.cc:2454] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 8.
I20260925 03:05:11.058108 27434 consensus_queue.cc:1035] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.066s
W20260925 03:05:11.146782 25163 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 174574us
I20260925 03:05:11.118396 26756 raft_consensus.cc:686] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [term 9 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:05:11.149816 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c81a1b3b616492ab6c964f45af36077" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:05:11.165952 25197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4ebd2b043024a548005c8b6b00b0449" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:05:11.118633 27230 consensus_queue.cc:1035] T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.037s
I20260925 03:05:11.167332 27273 leader_election.cc:288] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:11.167971 26756 consensus_queue.cc:237] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:05:11.187805 24621 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38570 (request call id 1655) took 1508 ms (client timeout 1531 ms). Trace:
W20260925 03:05:11.188309 24621 rpcz_store.cc:269] 0925 03:05:09.679846 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:05:09.679970 (+ 124us) service_pool.cc:227] Handling call
0925 03:05:11.187589 (+1507619us) spinlock_profiling.cc:243] Waited 35.3 ms on lock 0x7b54002193d4. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d42ae 00007f437b411c27 00007f4374af9c04 00007f4374af9b92 00007f4374af9ac1 00007f4374af9a3c 00007f4374af8cf2 00007f43748fa94c 00007f43748f9d86 00007f43748f972f
0925 03:05:11.187733 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":81235200}
I20260925 03:05:11.158696 27399 leader_election.cc:288] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 7 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:11.160025 27429 raft_consensus.cc:482] T 7e49248643df4131a1c42e74244e05c2 P 1bc1f2e62ab44af995165bd5116a9d43 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:11.174407 27384 leader_election.cc:288] T 7993f9c3e2a44f408bbf5d1e395e2306 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 7 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:05:11.146559 27370 consensus_queue.cc:1035] T 43366a0a2f9d4344bf4e37ffd290ef8a P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.128s
W20260925 03:05:11.220233 24436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 185949us
I20260925 03:05:11.169641 27379 consensus_queue.cc:237] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:05:11.225203 25163 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 59403.8us
I20260925 03:05:11.225281 24436 leader_election.cc:302] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:05:11.232476 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60d45b7d467d4e1ab92367f6d7d52b15" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:11.237514 27369 raft_consensus.cc:482] T 8e5819c15eac4544a57e5ee6c9d4f117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:11.238015 27369 raft_consensus.cc:504] T 8e5819c15eac4544a57e5ee6c9d4f117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:05:11.244246 26970 consensus_meta.cc:287] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c: Time spent flushing consensus metadata: real 0.599s user 0.010s sys 0.000s
I20260925 03:05:11.244671 26970 raft_consensus.cc:504] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:11.213706 27429 raft_consensus.cc:504] T 7e49248643df4131a1c42e74244e05c2 P 1bc1f2e62ab44af995165bd5116a9d43 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
W20260925 03:05:11.246745 27448 log.cc:924] Time spent T 11bb46bce3564841b7674e6a693bd2e6 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.125s user 0.000s sys 0.000s
I20260925 03:05:11.220099 27267 raft_consensus.cc:482] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
W20260925 03:05:11.367852 27438 log.cc:924] Time spent T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.376s user 0.001s sys 0.000s
I20260925 03:05:11.299880 24541 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "04ac285175464fbda67211d9574db415"
dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
from {username='slave'} at 127.0.0.1:36786
I20260925 03:05:11.313135 24542 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 1433) took 1579 ms. Trace:
I20260925 03:05:11.373250 24541 raft_consensus.cc:482] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 03:05:11.373811 24541 raft_consensus.cc:3037] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Advancing to term 8
I20260925 03:05:11.375656 24621 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a9c33fd100ff44c1a3247319a324a044"
dest_uuid: "eeab93abfdb141a895002ec9f312296c"
mode: GRACEFUL
new_leader_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
from {username='slave'} at 127.0.0.1:44570
I20260925 03:05:11.376609 24621 raft_consensus.cc:595] T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c [term 8 LEADER]: Received request to transfer leadership to TS a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:05:11.248812 24619 raft_consensus.cc:1226] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [term 10 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 9. Current term is 10. Ops: [9.5-9.5]
I20260925 03:05:11.278126 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
W20260925 03:05:11.387166 24436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 161970us
I20260925 03:05:11.388337 24436 leader_election.cc:302] T 442d4a29b1d743599f4b6b4871e26668 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:05:11.317998 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c88cbd39e01f44628a094af516be2833" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:11.390460 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4ebd2b043024a548005c8b6b00b0449" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:11.395504 24473 raft_consensus.cc:1226] T 11bb46bce3564841b7674e6a693bd2e6 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 4. Current term is 5. Ops: []
I20260925 03:05:11.365293 27456 raft_consensus.cc:482] T 0be4e0aa984f4f2b81a58e382e434ad5 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:11.365384 27428 raft_consensus.cc:482] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:05:11.277801 27273 raft_consensus.cc:482] T ac9357d144aa49789faaf31fe5e6bf89 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:11.399024 24473 raft_consensus.cc:1226] T 25102ba08e4e49388f6176badcd10a25 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 6. Current term is 7. Ops: []
I20260925 03:05:11.403410 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e5819c15eac4544a57e5ee6c9d4f117" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:11.248072 27429 leader_election.cc:288] T 7e49248643df4131a1c42e74244e05c2 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:11.414532 27273 raft_consensus.cc:504] T ac9357d144aa49789faaf31fe5e6bf89 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:11.412581 24464 raft_consensus.cc:2454] T 8e5819c15eac4544a57e5ee6c9d4f117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 7.
W20260925 03:05:11.417204 27427 consensus_meta.cc:287] T ceab6b0fccd54e909a64f66fe81a9a1f P 869b0a8575b54d08ae2817fad5dd42d5: Time spent flushing consensus metadata: real 0.690s user 0.014s sys 0.000s
I20260925 03:05:11.417622 27427 raft_consensus.cc:504] T ceab6b0fccd54e909a64f66fe81a9a1f P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:11.418686 24470 raft_consensus.cc:1226] T ceab6b0fccd54e909a64f66fe81a9a1f P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 8. Current term is 9. Ops: [8.4-8.4]
I20260925 03:05:11.275930 27267 raft_consensus.cc:504] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:11.409686 27456 raft_consensus.cc:504] T 0be4e0aa984f4f2b81a58e382e434ad5 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:11.422099 27456 leader_election.cc:288] T 0be4e0aa984f4f2b81a58e382e434ad5 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:11.423300 27077 raft_consensus.cc:504] T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:11.425489 27077 leader_election.cc:288] T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:11.408231 27431 raft_consensus.cc:2781] T 442d4a29b1d743599f4b6b4871e26668 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Leader election won for term 8
I20260925 03:05:11.411326 27440 raft_consensus.cc:482] T 48ca6ccd98194cf2af7b5278d15bf4ea P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:11.392974 27378 raft_consensus.cc:482] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:11.426190 24467 raft_consensus.cc:1226] T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 6. Current term is 7. Ops: [6.5-6.5]
I20260925 03:05:11.241226 27369 leader_election.cc:288] T 8e5819c15eac4544a57e5ee6c9d4f117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:11.310750 27432 raft_consensus.cc:2781] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260925 03:05:11.373558 24542 rpcz_store.cc:276] 0925 03:05:09.733622 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:05:09.733773 (+ 151us) service_pool.cc:227] Handling call
0925 03:05:09.734116 (+ 343us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:05:09.758406 (+ 24290us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:05:11.312889 (+1554483us) spinlock_profiling.cc:243] Waited 1.29 ms on lock 0x7b540012d754. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d0f51 00007f43764cf99e 00007f437b4112e9 00007f4374af7684 00007f4374af7612 00007f4374af7541 00007f4374af74bc 00007f4374af6772 00007f43748fa94c 00007f43748f9d86
0925 03:05:11.313072 (+ 183us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2961152}
I20260925 03:05:11.246793 26970 leader_election.cc:288] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 10 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:05:11.429275 24613 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c88cbd39e01f44628a094af516be2833"
dest_uuid: "eeab93abfdb141a895002ec9f312296c"
mode: GRACEFUL
new_leader_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
from {username='slave'} at 127.0.0.1:44570
I20260925 03:05:11.429997 24437 leader_election.cc:302] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:05:11.433542 24469 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1514) took 1431 ms. Trace:
I20260925 03:05:11.471804 24469 rpcz_store.cc:276] 0925 03:05:10.002307 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:05:10.010189 (+ 7882us) service_pool.cc:227] Handling call
0925 03:05:11.433340 (+1423151us) spinlock_profiling.cc:243] Waited 515 ms on lock 0x7b5400191ed4. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d42ae 00007f437b411c27 00007f4374af9c04 00007f4374af9b92 00007f4374af9ac1 00007f4374af9a3c 00007f4374af8cf2 00007f43748fa94c 00007f43748f9d86 00007f43748f972f
0925 03:05:11.433478 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1183871232}
I20260925 03:05:11.315594 26800 raft_consensus.cc:482] T 367277daf8d04778b3fa4ed94d71580d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:11.458961 27279 raft_consensus.cc:482] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:11.437081 24541 raft_consensus.cc:504] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:11.401127 24436 leader_election.cc:302] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:11.454239 24588 leader_election.cc:302] T a7aa47c48404465b94a36fd819fe3c7a P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:05:11.482843 24543 raft_consensus.cc:1226] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 7. Current term is 8. Ops: []
I20260925 03:05:11.317584 27402 consensus_queue.cc:1035] T 43366a0a2f9d4344bf4e37ffd290ef8a P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.124s
I20260925 03:05:11.460556 27432 raft_consensus.cc:482] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:05:11.487622 27414 raft_consensus.cc:482] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:11.522877 24613 raft_consensus.cc:595] T c88cbd39e01f44628a094af516be2833 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Received request to transfer leadership to TS a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:05:11.523355 24613 raft_consensus.cc:603] T c88cbd39e01f44628a094af516be2833 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260925 03:05:11.434427 27377 consensus_queue.cc:1046] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.052s
I20260925 03:05:11.473160 26800 raft_consensus.cc:504] T 367277daf8d04778b3fa4ed94d71580d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:11.491070 27241 raft_consensus.cc:2781] T a7aa47c48404465b94a36fd819fe3c7a P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Leader election won for term 8
I20260925 03:05:11.548280 27273 leader_election.cc:288] T ac9357d144aa49789faaf31fe5e6bf89 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:11.453533 27291 raft_consensus.cc:504] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:11.523958 27432 raft_consensus.cc:3037] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Advancing to term 10
I20260925 03:05:11.453318 27440 raft_consensus.cc:504] T 48ca6ccd98194cf2af7b5278d15bf4ea P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:11.524152 27463 raft_consensus.cc:482] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:11.426888 27428 raft_consensus.cc:504] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:11.571262 24541 leader_election.cc:288] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 8 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:11.572397 27428 leader_election.cc:288] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
W20260925 03:05:11.573700 27455 log.cc:924] Time spent T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.174s user 0.000s sys 0.000s
I20260925 03:05:11.473549 27279 raft_consensus.cc:504] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:11.576406 27267 leader_election.cc:288] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:05:11.569770 27463 raft_consensus.cc:504] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:11.585551 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7993f9c3e2a44f408bbf5d1e395e2306" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
W20260925 03:05:11.589282 24511 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.4:43979, user_credentials={real_user=slave}} blocked reactor thread for 148414us
I20260925 03:05:11.590525 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5f9b503b0ab43999d582015a367441e" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:05:11.591548 25196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "293ef69caf514ac98510a68495f54eda" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 8 candidate_status { last_received { term: 6 index: 9 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:05:11.592365 24514 leader_election.cc:398] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 7 election: Vote denied by peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187) with higher term. Message: Invalid argument: T 7b8e0d260a884ecdb20f79de46c36b19 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 7. Current term is 8.
I20260925 03:05:11.592722 24514 leader_election.cc:401] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 7 election: Cancelling election due to peer responding with higher term
I20260925 03:05:11.462347 27378 raft_consensus.cc:504] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:05:11.595657 24436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 194618us
I20260925 03:05:11.559716 27456 raft_consensus.cc:482] T 442d4a29b1d743599f4b6b4871e26668 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:11.596405 24436 leader_election.cc:302] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:11.596765 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74443a0a1dc34d88a2f42e3b722db405" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:05:11.597084 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7993f9c3e2a44f408bbf5d1e395e2306" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:11.597612 24473 raft_consensus.cc:3037] T 7993f9c3e2a44f408bbf5d1e395e2306 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:05:11.598810 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60d45b7d467d4e1ab92367f6d7d52b15" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:11.599692 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c88cbd39e01f44628a094af516be2833" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:11.602113 24615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5aec38e66a947c681e6c2a72b05e6ca" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:11.602504 24514 leader_election.cc:302] T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:05:11.568579 27053 raft_consensus.cc:3032] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [term 8 LEADER]: Stepping down as leader of term 8
I20260925 03:05:11.603688 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5f9b503b0ab43999d582015a367441e" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:11.611137 24471 raft_consensus.cc:1261] T 11bb46bce3564841b7674e6a693bd2e6 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260925 03:05:11.612264 24471 raft_consensus.cc:1261] T 25102ba08e4e49388f6176badcd10a25 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260925 03:05:11.615252 24473 raft_consensus.cc:2454] T 7993f9c3e2a44f408bbf5d1e395e2306 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 7.
I20260925 03:05:11.591048 27279 leader_election.cc:288] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
W20260925 03:05:11.621814 27468 log.cc:924] Time spent T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.061s user 0.000s sys 0.000s
I20260925 03:05:11.626457 26800 leader_election.cc:288] T 367277daf8d04778b3fa4ed94d71580d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:05:11.628180 27291 leader_election.cc:288] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 8 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:05:11.603395 26946 raft_consensus.cc:482] T 293ef69caf514ac98510a68495f54eda P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260925 03:05:11.629745 27467 log.cc:924] Time spent T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.107s user 0.000s sys 0.000s
I20260925 03:05:11.629447 26946 raft_consensus.cc:504] T 293ef69caf514ac98510a68495f54eda P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
W20260925 03:05:11.630486 27415 log.cc:924] Time spent T e6b6ec010caf4689b8c90449764f5cf9 P 869b0a8575b54d08ae2817fad5dd42d5: Append to log took a long time: real 0.187s user 0.001s sys 0.000s
I20260925 03:05:11.631676 26946 leader_election.cc:288] T 293ef69caf514ac98510a68495f54eda P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:05:11.633697 27378 leader_election.cc:288] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:05:11.609200 27053 raft_consensus.cc:729] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [term 8 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:05:11.634917 27299 raft_consensus.cc:2726] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Leader election lost for term 7. Reason: Vote denied by peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187) with higher term. Message: Invalid argument: T 7b8e0d260a884ecdb20f79de46c36b19 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 7. Current term is 8.
I20260925 03:05:11.635826 27299 consensus_queue.cc:1046] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260925 03:05:11.638222 24465 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38920 (request call id 210) took 1941 ms. Trace:
I20260925 03:05:11.638671 24465 rpcz_store.cc:276] 0925 03:05:09.696664 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:05:09.698704 (+ 2040us) service_pool.cc:227] Handling call
0925 03:05:09.731281 (+ 32577us) raft_consensus.cc:1457] Updating replica for 2 ops
0925 03:05:09.769459 (+ 38178us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:05:09.769489 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0925 03:05:09.812901 (+ 43412us) log.cc:842] Serialized 52 byte log entry
0925 03:05:09.992824 (+179923us) raft_consensus.cc:1631] Marking committed up to 8
0925 03:05:10.040620 (+ 47796us) log.cc:842] Serialized 14 byte log entry
0925 03:05:10.040836 (+ 216us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:05:10.040902 (+ 66us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:05:11.365874 (+1324972us) spinlock_profiling.cc:243] Waited 15.5 ms on lock 0x7b18003858a0. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f437c4478b1 00007f437c454661 00007f43735a8638 00007f43735abb11 00007f43735abac9 00007f43735aba59 00007f43735aba21 00007f43735aad1d 00007f437c80c204 00007f437c80c039 00007f437358adc6
0925 03:05:11.637858 (+271984us) raft_consensus.cc:1671] finished
0925 03:05:11.637989 (+ 131us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:05:11.638152 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":58638,"prepare.queue_time_us":48395,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":57,"raft.queue_time_us":35457,"raft.run_cpu_time_us":212,"raft.run_wall_time_us":8235,"spinlock_wait_cycles":35730304,"thread_start_us":148347,"threads_started":1,"wal-append.queue_time_us":286366}
I20260925 03:05:11.641263 27427 leader_election.cc:288] T ceab6b0fccd54e909a64f66fe81a9a1f P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 9 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
W20260925 03:05:11.659466 24547 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36786 (request call id 1435) took 1910 ms (client timeout 1929 ms). Trace:
W20260925 03:05:11.659981 24547 rpcz_store.cc:269] 0925 03:05:09.749463 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:05:09.754146 (+ 4683us) service_pool.cc:227] Handling call
0925 03:05:11.659220 (+1905074us) spinlock_profiling.cc:243] Waited 8.25 ms on lock 0x7b54004047d4. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d42ae 00007f437b411c27 00007f4374af9c04 00007f4374af9b92 00007f4374af9ac1 00007f4374af9a3c 00007f4374af8cf2 00007f43748fa94c 00007f43748f9d86 00007f43748f972f
0925 03:05:11.659387 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":18973056}
I20260925 03:05:11.677019 24547 raft_consensus.cc:3032] T 11bb46bce3564841b7674e6a693bd2e6 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 LEADER]: Stepping down as leader of term 4
I20260925 03:05:11.679382 24547 raft_consensus.cc:729] T 11bb46bce3564841b7674e6a693bd2e6 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:05:11.593056 27241 raft_consensus.cc:686] T a7aa47c48404465b94a36fd819fe3c7a P eeab93abfdb141a895002ec9f312296c [term 8 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
W20260925 03:05:11.642057 27325 consensus_meta.cc:287] T 74443a0a1dc34d88a2f42e3b722db405 P eeab93abfdb141a895002ec9f312296c: Time spent flushing consensus metadata: real 0.537s user 0.011s sys 0.000s
I20260925 03:05:11.642582 27272 raft_consensus.cc:2781] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260925 03:05:11.680140 24547 consensus_queue.cc:260] T 11bb46bce3564841b7674e6a693bd2e6 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:11.617894 27273 raft_consensus.cc:2726] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260925 03:05:11.676493 26800 raft_consensus.cc:2781] T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Leader election won for term 8
I20260925 03:05:11.553695 27414 raft_consensus.cc:504] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:11.712512 24547 raft_consensus.cc:3037] T 11bb46bce3564841b7674e6a693bd2e6 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:05:11.610170 27456 raft_consensus.cc:504] T 442d4a29b1d743599f4b6b4871e26668 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:05:11.696408 27461 log.cc:924] Time spent T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.321s user 0.001s sys 0.000s
I20260925 03:05:11.730309 27451 raft_consensus.cc:482] T b627ecb4a5384ef392aa695e06599c39 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:05:11.733161 27432 raft_consensus.cc:504] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:11.685942 26946 raft_consensus.cc:2726] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260925 03:05:11.681288 27299 raft_consensus.cc:3032] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 LEADER]: Stepping down as leader of term 7
W20260925 03:05:11.670197 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 67675us
I20260925 03:05:11.710600 27272 raft_consensus.cc:482] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260925 03:05:11.736752 27368 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.658s user 0.038s sys 0.042s
I20260925 03:05:11.783869 24547 raft_consensus.cc:1261] T 11bb46bce3564841b7674e6a693bd2e6 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20260925 03:05:11.784638 24547 pending_rounds.cc:76] T 11bb46bce3564841b7674e6a693bd2e6 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Aborting all ops after (but not including) 2
W20260925 03:05:11.784637 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 354725us
I20260925 03:05:11.785204 24547 pending_rounds.cc:97] T 11bb46bce3564841b7674e6a693bd2e6 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20260925 03:05:11.785485 24547 raft_consensus.cc:2866] T 11bb46bce3564841b7674e6a693bd2e6 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260925 03:05:11.787436 24436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 191116us
I20260925 03:05:11.697659 27325 raft_consensus.cc:504] T 74443a0a1dc34d88a2f42e3b722db405 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:05:11.784466 27368 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.715s user 0.038s sys 0.042s
I20260925 03:05:11.782960 24514 leader_election.cc:302] T b4ebd2b043024a548005c8b6b00b0449 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:05:11.745715 26800 raft_consensus.cc:686] T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 LEADER]: Becoming Leader. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:05:11.790759 24464 raft_consensus.cc:2379] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c in current term 10: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:05:11.795655 26946 raft_consensus.cc:482] T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:11.707832 27462 consensus_queue.cc:1046] T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.106s
I20260925 03:05:11.799364 24436 leader_election.cc:302] T 7993f9c3e2a44f408bbf5d1e395e2306 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:11.801318 24514 leader_election.cc:302] T 8e5819c15eac4544a57e5ee6c9d4f117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:05:11.786324 27272 raft_consensus.cc:3037] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Advancing to term 9
I20260925 03:05:11.782336 27299 raft_consensus.cc:729] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
W20260925 03:05:11.816669 27471 log.cc:924] Time spent T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5: Append to log took a long time: real 0.070s user 0.000s sys 0.001s
I20260925 03:05:11.825893 25197 raft_consensus.cc:1226] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43 [term 7 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 6. Current term is 7. Ops: [6.8-6.8]
I20260925 03:05:11.830719 24514 leader_election.cc:398] T b030d84c670b48d5bff337ccc0c77626 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 9 pre-election: Vote denied by peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187) with higher term. Message: Invalid argument: T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 9. Current term is 10.
I20260925 03:05:11.836424 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "293ef69caf514ac98510a68495f54eda" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:05:11.838282 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:11.836876 24473 raft_consensus.cc:1226] T a7aa47c48404465b94a36fd819fe3c7a P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 7. Current term is 8. Ops: []
I20260925 03:05:11.778395 27451 raft_consensus.cc:504] T b627ecb4a5384ef392aa695e06599c39 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:11.838963 24547 raft_consensus.cc:2454] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 8.
I20260925 03:05:11.842692 24615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e8af2641e124f9589c5ccfdac785f63" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:05:11.843202 24615 raft_consensus.cc:3037] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Advancing to term 8
I20260925 03:05:11.843452 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b8e0d260a884ecdb20f79de46c36b19" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:05:11.841161 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c81a1b3b616492ab6c964f45af36077" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:11.844565 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48ca6ccd98194cf2af7b5278d15bf4ea" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:11.846017 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e5819c15eac4544a57e5ee6c9d4f117" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:11.804533 27481 raft_consensus.cc:2726] T b4ebd2b043024a548005c8b6b00b0449 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260925 03:05:11.807776 27273 raft_consensus.cc:2726] T 7993f9c3e2a44f408bbf5d1e395e2306 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260925 03:05:11.856348 24621 raft_consensus.cc:1226] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 7. Current term is 8. Ops: []
I20260925 03:05:11.850344 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ac285175464fbda67211d9574db415" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: true dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:05:11.859299 24620 raft_consensus.cc:3032] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [term 7 LEADER]: Stepping down as leader of term 7
I20260925 03:05:11.859620 24620 raft_consensus.cc:729] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [term 7 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:05:11.852911 24588 leader_election.cc:302] T c88cbd39e01f44628a094af516be2833 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:05:11.833760 27279 raft_consensus.cc:3032] T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 LEADER]: Stepping down as leader of term 8
I20260925 03:05:11.849969 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "420163a6cdce412ebc21c25fe3835638" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:11.830344 27431 raft_consensus.cc:686] T 442d4a29b1d743599f4b6b4871e26668 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 LEADER]: Becoming Leader. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:05:11.857033 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be4e0aa984f4f2b81a58e382e434ad5" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:11.804318 27477 consensus_queue.cc:1046] T 25102ba08e4e49388f6176badcd10a25 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.033s
I20260925 03:05:11.837190 27368 raft_consensus.cc:2781] T 8e5819c15eac4544a57e5ee6c9d4f117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260925 03:05:11.833019 27077 raft_consensus.cc:482] T 4e5332a5d329493991dff994e70ea92c P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:11.834331 27440 leader_election.cc:288] T 48ca6ccd98194cf2af7b5278d15bf4ea P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:11.867906 24615 raft_consensus.cc:2454] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 8.
I20260925 03:05:11.860340 24620 consensus_queue.cc:260] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:11.813851 26946 raft_consensus.cc:504] T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:11.787144 27241 consensus_queue.cc:237] T a7aa47c48404465b94a36fd819fe3c7a P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:12.479184 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5f9b503b0ab43999d582015a367441e" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
W20260925 03:05:12.257014 25165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 418754us
I20260925 03:05:12.481007 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9c33fd100ff44c1a3247319a324a044" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:12.481271 25197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367277daf8d04778b3fa4ed94d71580d" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:05:12.481989 25197 raft_consensus.cc:2454] T 367277daf8d04778b3fa4ed94d71580d P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 6.
I20260925 03:05:12.482267 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "550118bc1a0641ce9803bdbf43f2069f" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:12.483412 25197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "293ef69caf514ac98510a68495f54eda" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 8 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:05:12.114053 24465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e8af2641e124f9589c5ccfdac785f63" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:12.483412 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9357d144aa49789faaf31fe5e6bf89" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:12.491677 24465 raft_consensus.cc:2379] T 9e8af2641e124f9589c5ccfdac785f63 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c in current term 8: Already voted for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in this term.
I20260925 03:05:12.370563 24468 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1436) took 1736 ms. Trace:
I20260925 03:05:12.493554 24615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be4e0aa984f4f2b81a58e382e434ad5" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:11.847254 27407 raft_consensus.cc:482] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:05:11.884519 24469 raft_consensus.cc:2454] T 0be4e0aa984f4f2b81a58e382e434ad5 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 7.
I20260925 03:05:12.490242 24620 raft_consensus.cc:3037] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Advancing to term 8
W20260925 03:05:11.992913 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 313110us
W20260925 03:05:11.894862 24622 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38570 (request call id 1662) took 2091 ms (client timeout 1649 ms). Trace:
I20260925 03:05:12.168061 24466 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55560 (request call id 1522) took 1165 ms. Trace:
W20260925 03:05:12.479416 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 626569us
I20260925 03:05:11.849596 27470 raft_consensus.cc:482] T edcf6e6a422d4d198d4fbb0f7d3d845d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
W20260925 03:05:11.927126 27472 log.cc:924] Time spent T c88cbd39e01f44628a094af516be2833 P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.213s user 0.000s sys 0.001s
I20260925 03:05:12.493031 24468 rpcz_store.cc:276] 0925 03:05:10.633708 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:05:10.758679 (+124971us) service_pool.cc:227] Handling call
0925 03:05:10.758973 (+ 294us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:05:10.978059 (+219086us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:05:12.370217 (+1392158us) spinlock_profiling.cc:243] Waited 217 ms on lock 0x7b54001928d4. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d0f51 00007f43764cf99e 00007f437b4112e9 00007f4374af7684 00007f4374af7612 00007f4374af7541 00007f4374af74bc 00007f4374af6772 00007f43748fa94c 00007f43748f9d86
0925 03:05:12.370481 (+ 264us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":499876096}
W20260925 03:05:12.478966 27484 log.cc:924] Time spent T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.262s user 0.000s sys 0.000s
I20260925 03:05:12.514776 24466 rpcz_store.cc:276] 0925 03:05:11.002742 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:05:11.002876 (+ 134us) service_pool.cc:227] Handling call
0925 03:05:11.025392 (+ 22516us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:05:11.025756 (+ 364us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:05:11.025784 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:05:11.221099 (+195315us) log.cc:842] Serialized 26 byte log entry
0925 03:05:11.490767 (+269668us) raft_consensus.cc:1631] Marking committed up to 3
0925 03:05:11.490803 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:05:11.490875 (+ 72us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:05:12.167770 (+676895us) raft_consensus.cc:1671] finished
0925 03:05:12.167796 (+ 26us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:05:12.167982 (+ 186us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":248,"raft.run_cpu_time_us":154,"raft.run_wall_time_us":16101,"thread_start_us":153434,"threads_started":1,"wal-append.queue_time_us":402290}
I20260925 03:05:12.518959 25165 leader_election.cc:302] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
W20260925 03:05:12.514196 24622 rpcz_store.cc:269] 0925 03:05:09.803775 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:05:09.804973 (+ 1198us) service_pool.cc:227] Handling call
0925 03:05:11.849822 (+2044849us) spinlock_profiling.cc:243] Waited 5.68 ms on lock 0x7b58001d3d98. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f43733ecf11 00007f4373570851 00007f4373570702 00007f437c460b09 00007f437b39c730 00007f437b46279b 00007f437b462821 00007f437b4121b0 00007f437b411a94 00007f4374af9c04 00007f4374af9b92
0925 03:05:11.894765 (+ 44943us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":13064064}
I20260925 03:05:11.870270 27279 raft_consensus.cc:729] T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
W20260925 03:05:11.940265 27475 log.cc:924] Time spent T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.092s user 0.000s sys 0.000s
I20260925 03:05:12.527509 24473 raft_consensus.cc:2454] T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 6.
I20260925 03:05:12.452076 27325 leader_election.cc:288] T 74443a0a1dc34d88a2f42e3b722db405 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 7 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:05:12.521226 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e725b82985746b39c0cc8f671c95353" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:12.523610 24615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367277daf8d04778b3fa4ed94d71580d" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:12.492673 27368 raft_consensus.cc:482] T 8e5819c15eac4544a57e5ee6c9d4f117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260925 03:05:11.930085 27483 log.cc:924] Time spent T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5: Append to log took a long time: real 0.068s user 0.000s sys 0.000s
I20260925 03:05:12.150219 27273 raft_consensus.cc:482] T edcf6e6a422d4d198d4fbb0f7d3d845d P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:12.460559 27485 raft_consensus.cc:482] T f22ddd7d1c444dc58f45a1fb0b3e02bc P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:11.918154 27253 raft_consensus.cc:2726] T c88cbd39e01f44628a094af516be2833 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260925 03:05:11.926301 27456 leader_election.cc:288] T 442d4a29b1d743599f4b6b4871e26668 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:11.936305 27451 leader_election.cc:288] T b627ecb4a5384ef392aa695e06599c39 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:12.520773 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74443a0a1dc34d88a2f42e3b722db405" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:05:12.572000 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "293ef69caf514ac98510a68495f54eda" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 8 candidate_status { last_received { term: 6 index: 9 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:12.572866 24468 raft_consensus.cc:2454] T 293ef69caf514ac98510a68495f54eda P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 7.
I20260925 03:05:12.572643 24621 raft_consensus.cc:2379] T 74443a0a1dc34d88a2f42e3b722db405 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 in current term 7: Already voted for candidate eeab93abfdb141a895002ec9f312296c in this term.
I20260925 03:05:12.092996 27463 leader_election.cc:288] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:12.530619 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367277daf8d04778b3fa4ed94d71580d" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:12.541320 24465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:11.926013 27414 leader_election.cc:288] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:05:12.193251 27486 raft_consensus.cc:482] T c85f5d1d3f004cf5817bb594fd8e6469 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:12.551295 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ac285175464fbda67211d9574db415" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: true dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:12.479257 27432 leader_election.cc:288] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 10 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:12.550840 24615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60d45b7d467d4e1ab92367f6d7d52b15" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:05:12.557103 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceab6b0fccd54e909a64f66fe81a9a1f" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:05:12.574218 24466 raft_consensus.cc:3037] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Advancing to term 8
I20260925 03:05:12.574447 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f1fac028a144058b92331c0d9b44ebf" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:12.574836 24615 raft_consensus.cc:3037] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Advancing to term 10
I20260925 03:05:12.575781 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:12.576496 24468 raft_consensus.cc:2454] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 8.
I20260925 03:05:12.552563 27368 raft_consensus.cc:3037] T 8e5819c15eac4544a57e5ee6c9d4f117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Advancing to term 9
I20260925 03:05:12.578454 24620 raft_consensus.cc:2454] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 8.
I20260925 03:05:12.579509 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5aec38e66a947c681e6c2a72b05e6ca" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:12.590704 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b627ecb4a5384ef392aa695e06599c39" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:12.584573 24615 raft_consensus.cc:2454] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 10.
W20260925 03:05:12.587387 24511 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.993s user 0.020s sys 0.041s
W20260925 03:05:12.590819 24436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 78689.5us
I20260925 03:05:12.545037 27494 raft_consensus.cc:2726] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260925 03:05:12.509711 27077 raft_consensus.cc:504] T 4e5332a5d329493991dff994e70ea92c P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:12.592672 24621 raft_consensus.cc:2379] T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 in current term 8: Already voted for candidate eeab93abfdb141a895002ec9f312296c in this term.
I20260925 03:05:12.593223 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442d4a29b1d743599f4b6b4871e26668" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:12.607962 24622 raft_consensus.cc:1226] T 25102ba08e4e49388f6176badcd10a25 P eeab93abfdb141a895002ec9f312296c [term 7 LEADER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 6. Current term is 7. Ops: []
W20260925 03:05:12.595849 24511 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.999s user 0.020s sys 0.041s
I20260925 03:05:12.592751 24614 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38564 (request call id 1491) took 1833 ms. Trace:
I20260925 03:05:11.983814 26800 consensus_queue.cc:237] T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:12.614233 24614 rpcz_store.cc:276] 0925 03:05:10.759389 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:05:11.016628 (+257239us) service_pool.cc:227] Handling call
0925 03:05:11.043642 (+ 27014us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:05:11.052505 (+ 8863us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:05:11.052540 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:05:11.223717 (+171177us) log.cc:842] Serialized 26 byte log entry
0925 03:05:11.522694 (+298977us) raft_consensus.cc:1631] Marking committed up to 3
0925 03:05:11.522731 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:05:11.522820 (+ 89us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:05:12.592530 (+1069710us) raft_consensus.cc:1671] finished
0925 03:05:12.592549 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:05:12.592666 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13132,"raft.queue_time_us":8754,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":11808,"thread_start_us":298676,"threads_started":1,"wal-append.queue_time_us":419478}
W20260925 03:05:12.615927 24511 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 1.02155e+06us
I20260925 03:05:11.937894 27431 consensus_queue.cc:237] T 442d4a29b1d743599f4b6b4871e26668 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:12.624014 24511 leader_election.cc:302] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:12.605594 24466 raft_consensus.cc:2454] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 8.
I20260925 03:05:12.609027 24548 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 1483) took 1200 ms. Trace:
I20260925 03:05:12.522818 27407 raft_consensus.cc:504] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:12.624986 24548 rpcz_store.cc:276] 0925 03:05:11.408708 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:05:11.603782 (+195074us) service_pool.cc:227] Handling call
0925 03:05:11.604097 (+ 315us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:05:11.640680 (+ 36583us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:05:11.640736 (+ 56us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:05:11.641241 (+ 505us) log.cc:842] Serialized 26 byte log entry
0925 03:05:11.675945 (+ 34704us) raft_consensus.cc:1631] Marking committed up to 3
0925 03:05:11.675981 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:05:11.676166 (+ 185us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:05:12.608836 (+932670us) raft_consensus.cc:1671] finished
0925 03:05:12.608853 (+ 17us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:05:12.608959 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":57701,"raft.queue_time_us":65506,"raft.run_cpu_time_us":133,"raft.run_wall_time_us":10896,"thread_start_us":34231,"threads_started":1,"wal-append.queue_time_us":207232}
I20260925 03:05:12.578138 27496 raft_consensus.cc:482] T 0b5df303cd6a4d59aca5c18bbbff24ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:12.525686 27470 raft_consensus.cc:504] T edcf6e6a422d4d198d4fbb0f7d3d845d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:12.630614 24465 raft_consensus.cc:1226] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20260925 03:05:11.887506 27299 consensus_queue.cc:260] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:12.635254 24511 leader_election.cc:302] T 7993f9c3e2a44f408bbf5d1e395e2306 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:12.586807 27432 raft_consensus.cc:482] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:12.593711 24620 raft_consensus.cc:1226] T 11bb46bce3564841b7674e6a693bd2e6 P eeab93abfdb141a895002ec9f312296c [term 5 LEADER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 4. Current term is 5. Ops: []
I20260925 03:05:12.592095 27463 raft_consensus.cc:482] T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:12.638667 24548 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 1490) took 1197 ms. Trace:
I20260925 03:05:12.648152 24548 rpcz_store.cc:276] 0925 03:05:11.441686 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:05:12.637835 (+1196149us) service_pool.cc:227] Handling call
0925 03:05:12.638172 (+ 337us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:05:12.638346 (+ 174us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:05:12.638375 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:05:12.638414 (+ 39us) raft_consensus.cc:1631] Marking committed up to 3
0925 03:05:12.638444 (+ 30us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:05:12.638502 (+ 58us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:05:12.638607 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:05:12.627890 27451 raft_consensus.cc:482] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:12.579392 27273 raft_consensus.cc:504] T edcf6e6a422d4d198d4fbb0f7d3d845d P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:12.576709 27485 raft_consensus.cc:504] T f22ddd7d1c444dc58f45a1fb0b3e02bc P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:05:12.670014 25164 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.577s user 0.002s sys 0.006s
I20260925 03:05:12.591800 27486 raft_consensus.cc:504] T c85f5d1d3f004cf5817bb594fd8e6469 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:12.600595 27279 consensus_queue.cc:260] T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:12.671582 27497 raft_consensus.cc:2726] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
W20260925 03:05:12.696957 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 179778us
I20260925 03:05:12.672806 27477 raft_consensus.cc:482] T 420163a6cdce412ebc21c25fe3835638 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:12.658993 27432 raft_consensus.cc:504] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:12.658674 27463 raft_consensus.cc:504] T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:12.710403 24464 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 1453) took 1114 ms. Trace:
W20260925 03:05:12.697718 25164 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.809s user 0.004s sys 0.006s
W20260925 03:05:12.706737 24511 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 71557.8us
I20260925 03:05:12.711433 24464 rpcz_store.cc:276] 0925 03:05:11.596093 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:05:11.598581 (+ 2488us) service_pool.cc:227] Handling call
0925 03:05:12.710199 (+1111618us) spinlock_profiling.cc:243] Waited 191 ms on lock 0x7b5400090454. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d42ae 00007f437b411c27 00007f4374af9c04 00007f4374af9b92 00007f4374af9ac1 00007f4374af9a3c 00007f4374af8cf2 00007f43748fa94c 00007f43748f9d86 00007f43748f972f
0925 03:05:12.710348 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":439623424}
I20260925 03:05:12.690459 27495 raft_consensus.cc:482] T c85f5d1d3f004cf5817bb594fd8e6469 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:12.668314 27451 raft_consensus.cc:504] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:12.715816 24469 raft_consensus.cc:1226] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [term 10 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 9. Current term is 10. Ops: []
I20260925 03:05:12.716820 24511 leader_election.cc:302] T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 1bc1f2e62ab44af995165bd5116a9d43, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:12.657502 27492 consensus_queue.cc:1046] T a7aa47c48404465b94a36fd819fe3c7a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.071s
I20260925 03:05:12.727056 27492 raft_consensus.cc:3032] T a7aa47c48404465b94a36fd819fe3c7a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 LEADER]: Stepping down as leader of term 7
I20260925 03:05:12.727415 27492 raft_consensus.cc:729] T a7aa47c48404465b94a36fd819fe3c7a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
W20260925 03:05:12.730417 24436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.4:43979, user_credentials={real_user=slave}} blocked reactor thread for 55786.8us
I20260925 03:05:12.728480 27485 leader_election.cc:288] T f22ddd7d1c444dc58f45a1fb0b3e02bc P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:12.651000 27496 raft_consensus.cc:504] T 0b5df303cd6a4d59aca5c18bbbff24ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:12.738770 27496 leader_election.cc:288] T 0b5df303cd6a4d59aca5c18bbbff24ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:12.740774 24619 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38564 (request call id 1483) took 2978 ms. Trace:
I20260925 03:05:12.707726 27499 raft_consensus.cc:482] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:12.714679 27279 raft_consensus.cc:3037] T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Advancing to term 9
I20260925 03:05:12.741146 24619 rpcz_store.cc:276] 0925 03:05:09.762015 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:05:09.766127 (+ 4112us) service_pool.cc:227] Handling call
0925 03:05:11.188870 (+1422743us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:05:11.377595 (+188725us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:05:12.739723 (+1362128us) spinlock_profiling.cc:243] Waited 59.7 ms on lock 0x7b54002193d4. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d0f51 00007f43764cf99e 00007f437b4112e9 00007f4374af7684 00007f4374af7612 00007f4374af7541 00007f4374af74bc 00007f4374af6772 00007f43748fa94c 00007f43748f9d86
0925 03:05:12.740571 (+ 848us) spinlock_profiling.cc:243] Waited 1.42 s on lock 0x7b54002193d0. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764cfa09 00007f437b4112e9 00007f4374af7684 00007f4374af7612 00007f4374af7541 00007f4374af74bc 00007f4374af6772 00007f43748fa94c 00007f43748f9d86 00007f43748f972f
0925 03:05:12.740714 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3408478336}
I20260925 03:05:12.730546 27497 raft_consensus.cc:2726] T 7993f9c3e2a44f408bbf5d1e395e2306 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260925 03:05:12.738659 27432 leader_election.cc:288] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:12.706419 27477 raft_consensus.cc:504] T 420163a6cdce412ebc21c25fe3835638 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:12.765034 27477 leader_election.cc:288] T 420163a6cdce412ebc21c25fe3835638 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:12.745083 27498 raft_consensus.cc:482] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1bc1f2e62ab44af995165bd5116a9d43)
I20260925 03:05:12.746280 26946 leader_election.cc:288] T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:12.748133 27508 raft_consensus.cc:482] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:12.732220 27505 raft_consensus.cc:482] T ac9357d144aa49789faaf31fe5e6bf89 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:12.728272 27492 consensus_queue.cc:260] T a7aa47c48404465b94a36fd819fe3c7a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:05:12.773594 24511 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 56518us
I20260925 03:05:12.775058 24511 leader_election.cc:302] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 1bc1f2e62ab44af995165bd5116a9d43, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:12.730845 27495 raft_consensus.cc:504] T c85f5d1d3f004cf5817bb594fd8e6469 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:12.770246 27485 raft_consensus.cc:482] T 100c113a9fd84cc8adf2ec7b33ea0093 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:12.783856 27477 raft_consensus.cc:2726] T 71f461a6e2ff434993b78a2d077f0abd P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260925 03:05:12.789333 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 86840.2us
I20260925 03:05:12.765991 27376 consensus_queue.cc:1035] T 11bb46bce3564841b7674e6a693bd2e6 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.024s
I20260925 03:05:12.769338 27077 leader_election.cc:288] T 4e5332a5d329493991dff994e70ea92c P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:12.790668 24588 leader_election.cc:302] T 7993f9c3e2a44f408bbf5d1e395e2306 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
W20260925 03:05:12.793373 27490 log.cc:924] Time spent T 48ca6ccd98194cf2af7b5278d15bf4ea P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.090s user 0.000s sys 0.001s
I20260925 03:05:12.785231 27492 raft_consensus.cc:3037] T a7aa47c48404465b94a36fd819fe3c7a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Advancing to term 8
I20260925 03:05:12.771294 27432 consensus_queue.cc:1046] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.021s
I20260925 03:05:12.798115 27477 consensus_queue.cc:1035] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.007s
I20260925 03:05:12.805126 27509 raft_consensus.cc:2726] T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260925 03:05:12.805627 27486 leader_election.cc:288] T c85f5d1d3f004cf5817bb594fd8e6469 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
W20260925 03:05:12.806098 27504 log.cc:924] Time spent T 0be4e0aa984f4f2b81a58e382e434ad5 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.066s user 0.001s sys 0.000s
I20260925 03:05:12.807389 27495 leader_election.cc:288] T c85f5d1d3f004cf5817bb594fd8e6469 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:12.797561 27470 leader_election.cc:288] T edcf6e6a422d4d198d4fbb0f7d3d845d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:12.794140 27273 leader_election.cc:288] T edcf6e6a422d4d198d4fbb0f7d3d845d P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:12.831153 27368 raft_consensus.cc:504] T 8e5819c15eac4544a57e5ee6c9d4f117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:05:12.825053 27346 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.807s user 0.002s sys 0.021s
I20260925 03:05:12.795481 27485 raft_consensus.cc:504] T 100c113a9fd84cc8adf2ec7b33ea0093 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:12.753638 27499 raft_consensus.cc:504] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:12.775190 27508 raft_consensus.cc:504] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:12.822631 27501 consensus_queue.cc:1035] T 25102ba08e4e49388f6176badcd10a25 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.050s
W20260925 03:05:12.839255 27346 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.826s user 0.003s sys 0.022s
I20260925 03:05:12.780905 27505 raft_consensus.cc:504] T ac9357d144aa49789faaf31fe5e6bf89 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:12.837731 27407 leader_election.cc:288] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:05:12.852098 27487 raft_consensus.cc:2781] T 7993f9c3e2a44f408bbf5d1e395e2306 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Leader election won for term 7
I20260925 03:05:12.829011 27463 leader_election.cc:288] T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
W20260925 03:05:12.862980 27272 consensus_meta.cc:287] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5: Time spent flushing consensus metadata: real 1.027s user 0.012s sys 0.000s
I20260925 03:05:12.863494 27272 raft_consensus.cc:504] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:12.854097 27513 raft_consensus.cc:482] T 7b8e0d260a884ecdb20f79de46c36b19 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:12.771065 27498 raft_consensus.cc:504] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:12.893574 27487 raft_consensus.cc:686] T 7993f9c3e2a44f408bbf5d1e395e2306 P eeab93abfdb141a895002ec9f312296c [term 7 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:05:12.897119 27487 consensus_queue.cc:237] T 7993f9c3e2a44f408bbf5d1e395e2306 P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:12.902101 27499 leader_election.cc:288] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:12.903956 27485 leader_election.cc:288] T 100c113a9fd84cc8adf2ec7b33ea0093 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:12.904570 27513 raft_consensus.cc:504] T 7b8e0d260a884ecdb20f79de46c36b19 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:12.910207 27508 leader_election.cc:288] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
W20260925 03:05:12.970973 24436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.4:43979, user_credentials={real_user=slave}} blocked reactor thread for 239705us
I20260925 03:05:12.971859 24468 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38920 (request call id 220) took 1133 ms. Trace:
I20260925 03:05:12.972831 24468 rpcz_store.cc:276] 0925 03:05:11.837830 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:05:12.594953 (+757123us) service_pool.cc:227] Handling call
0925 03:05:12.595270 (+ 317us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:05:12.709978 (+114708us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:05:12.710011 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:05:12.710441 (+ 430us) log.cc:842] Serialized 26 byte log entry
0925 03:05:12.773147 (+ 62706us) raft_consensus.cc:1631] Marking committed up to 8
0925 03:05:12.773189 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:05:12.773321 (+ 132us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:05:12.970801 (+197480us) raft_consensus.cc:1671] finished
0925 03:05:12.970891 (+ 90us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:05:12.971507 (+ 616us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4669,"raft.queue_time_us":147849,"raft.run_cpu_time_us":138,"raft.run_wall_time_us":7087,"thread_start_us":151147,"threads_started":2,"wal-append.queue_time_us":98573}
I20260925 03:05:12.982479 24539 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 1485) took 1573 ms. Trace:
I20260925 03:05:12.983681 24588 leader_election.cc:302] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:05:12.983093 24539 rpcz_store.cc:276] 0925 03:05:11.409242 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:05:11.731846 (+322604us) service_pool.cc:227] Handling call
0925 03:05:11.732206 (+ 360us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:05:11.736710 (+ 4504us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:05:11.736740 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:05:11.737138 (+ 398us) log.cc:842] Serialized 26 byte log entry
0925 03:05:11.997468 (+260330us) raft_consensus.cc:1631] Marking committed up to 3
0925 03:05:11.997512 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:05:11.997595 (+ 83us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:05:12.981539 (+983944us) raft_consensus.cc:1671] finished
0925 03:05:12.981584 (+ 45us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:05:12.982150 (+ 566us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4125,"raft.queue_time_us":5179,"raft.run_cpu_time_us":204,"raft.run_wall_time_us":34521,"thread_start_us":209750,"threads_started":1,"wal-append.queue_time_us":849599}
I20260925 03:05:12.986125 24547 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 1488) took 1576 ms. Trace:
I20260925 03:05:12.986696 24547 rpcz_store.cc:276] 0925 03:05:11.409970 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:05:12.098560 (+688590us) service_pool.cc:227] Handling call
0925 03:05:12.098991 (+ 431us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:05:12.099352 (+ 361us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:05:12.099380 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:05:12.493891 (+394511us) log.cc:842] Serialized 26 byte log entry
0925 03:05:12.644392 (+150501us) raft_consensus.cc:1631] Marking committed up to 4
0925 03:05:12.644428 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:05:12.644495 (+ 67us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:05:12.985427 (+340932us) raft_consensus.cc:1671] finished
0925 03:05:12.985479 (+ 52us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:05:12.985895 (+ 416us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":28655,"raft.queue_time_us":418345,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":2549,"thread_start_us":99000,"threads_started":1,"wal-append.queue_time_us":176111}
I20260925 03:05:12.991104 24539 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 1492) took 1548 ms. Trace:
I20260925 03:05:12.993072 24539 rpcz_store.cc:276] 0925 03:05:11.442197 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:05:12.987345 (+1545148us) service_pool.cc:227] Handling call
0925 03:05:12.988160 (+ 815us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:05:12.988679 (+ 519us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:05:12.988732 (+ 53us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:05:12.990761 (+ 2029us) raft_consensus.cc:1631] Marking committed up to 5
0925 03:05:12.990806 (+ 45us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:05:12.990895 (+ 89us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:05:12.991008 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:05:13.013175 27505 leader_election.cc:288] T ac9357d144aa49789faaf31fe5e6bf89 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:13.013870 24619 raft_consensus.cc:1261] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [term 10 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 10 index: 5. (index mismatch)
I20260925 03:05:13.015465 27272 leader_election.cc:288] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 9 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:13.017554 27341 raft_consensus.cc:482] T e92f377a79354a6ebb4eec67303c13e3 P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:13.023639 26762 consensus_queue.cc:1035] T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260925 03:05:13.027750 26762 raft_consensus.cc:482] T a4abe0ea757e4886be4945c5001e529a P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:05:13.028287 27040 raft_consensus.cc:2726] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260925 03:05:13.023567 27341 raft_consensus.cc:504] T e92f377a79354a6ebb4eec67303c13e3 P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:13.028667 26762 raft_consensus.cc:504] T a4abe0ea757e4886be4945c5001e529a P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:13.037204 27386 raft_consensus.cc:482] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1bc1f2e62ab44af995165bd5116a9d43)
I20260925 03:05:13.037129 27268 raft_consensus.cc:482] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:13.050321 27339 raft_consensus.cc:482] T 29637f5a4a504ce6ba29033ccd0cd716 P 1bc1f2e62ab44af995165bd5116a9d43 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:05:13.051231 27513 leader_election.cc:288] T 7b8e0d260a884ecdb20f79de46c36b19 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:13.051743 27339 raft_consensus.cc:504] T 29637f5a4a504ce6ba29033ccd0cd716 P 1bc1f2e62ab44af995165bd5116a9d43 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:13.054101 27498 leader_election.cc:288] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:05:13.057525 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e92f377a79354a6ebb4eec67303c13e3" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:13.058627 24539 raft_consensus.cc:2454] T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 6.
I20260925 03:05:13.058710 27339 leader_election.cc:288] T 29637f5a4a504ce6ba29033ccd0cd716 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
W20260925 03:05:13.033157 24511 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 52732.9us
I20260925 03:05:13.036465 27463 raft_consensus.cc:482] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:13.033874 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f22ddd7d1c444dc58f45a1fb0b3e02bc" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:13.033226 27341 leader_election.cc:288] T e92f377a79354a6ebb4eec67303c13e3 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:13.066777 27268 raft_consensus.cc:504] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:13.032335 27241 raft_consensus.cc:482] T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:05:13.029274 27376 raft_consensus.cc:482] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1bc1f2e62ab44af995165bd5116a9d43)
I20260925 03:05:13.068660 27241 raft_consensus.cc:504] T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:13.066007 24622 raft_consensus.cc:2454] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 8.
W20260925 03:05:13.083005 24437 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.692s user 0.024s sys 0.020s
I20260925 03:05:13.069473 27376 raft_consensus.cc:504] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
W20260925 03:05:13.084524 24437 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.700s user 0.024s sys 0.020s
I20260925 03:05:13.030313 27368 leader_election.cc:288] T 8e5819c15eac4544a57e5ee6c9d4f117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 9 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:13.089695 27513 raft_consensus.cc:482] T 9e8af2641e124f9589c5ccfdac785f63 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:13.093000 24436 leader_election.cc:302] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:13.104796 27485 raft_consensus.cc:482] T b4ebd2b043024a548005c8b6b00b0449 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1bc1f2e62ab44af995165bd5116a9d43)
I20260925 03:05:13.106519 27485 raft_consensus.cc:504] T b4ebd2b043024a548005c8b6b00b0449 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:13.054090 27493 raft_consensus.cc:482] T a4abe0ea757e4886be4945c5001e529a P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
W20260925 03:05:13.099195 25165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 70759.2us
I20260925 03:05:13.065400 27463 raft_consensus.cc:504] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:13.100838 24514 leader_election.cc:302] T 0be4e0aa984f4f2b81a58e382e434ad5 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:05:13.102065 27508 raft_consensus.cc:482] T e6b6ec010caf4689b8c90449764f5cf9 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 1bc1f2e62ab44af995165bd5116a9d43)
I20260925 03:05:13.103158 27346 raft_consensus.cc:482] T cc2df40db08649c9a75e90000b878559 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:13.093000 27513 raft_consensus.cc:504] T 9e8af2641e124f9589c5ccfdac785f63 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:13.113240 25165 leader_election.cc:302] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:13.113765 27463 leader_election.cc:288] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:13.038239 27386 raft_consensus.cc:504] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:13.114192 27513 leader_election.cc:288] T 9e8af2641e124f9589c5ccfdac785f63 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:13.116158 25197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:05:13.117302 27346 raft_consensus.cc:504] T cc2df40db08649c9a75e90000b878559 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:13.120482 27501 raft_consensus.cc:482] T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:13.109661 27485 leader_election.cc:288] T b4ebd2b043024a548005c8b6b00b0449 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:05:13.126673 25197 raft_consensus.cc:2396] T 29637f5a4a504ce6ba29033ccd0cd716 P 1bc1f2e62ab44af995165bd5116a9d43 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c for term 10 because replica has last-logged OpId of term: 9 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 6.
I20260925 03:05:13.131660 27463 raft_consensus.cc:2781] T 0be4e0aa984f4f2b81a58e382e434ad5 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260925 03:05:13.132231 27463 raft_consensus.cc:482] T 0be4e0aa984f4f2b81a58e382e434ad5 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:13.132761 27463 raft_consensus.cc:3037] T 0be4e0aa984f4f2b81a58e382e434ad5 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Advancing to term 8
I20260925 03:05:13.133297 27376 leader_election.cc:288] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:05:13.133924 24469 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55560 (request call id 1536) took 1294 ms. Trace:
I20260925 03:05:13.109314 27493 raft_consensus.cc:504] T a4abe0ea757e4886be4945c5001e529a P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:13.133626 27501 raft_consensus.cc:504] T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:13.135834 27346 leader_election.cc:288] T cc2df40db08649c9a75e90000b878559 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:13.131503 27529 raft_consensus.cc:2726] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260925 03:05:13.137215 27268 leader_election.cc:288] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:13.137729 24471 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1461) took 1539 ms. Trace:
I20260925 03:05:13.136641 27241 leader_election.cc:288] T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:05:13.140287 24436 leader_election.cc:302] T 550118bc1a0641ce9803bdbf43f2069f P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:13.141125 27521 raft_consensus.cc:482] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:13.145932 24543 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 1481) took 1738 ms. Trace:
I20260925 03:05:13.148182 27386 leader_election.cc:288] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:05:13.149718 24544 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 1595) took 3215 ms. Trace:
I20260925 03:05:13.150398 26762 leader_election.cc:288] T a4abe0ea757e4886be4945c5001e529a P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
W20260925 03:05:13.151363 24545 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33068 (request call id 200) took 3056 ms (client timeout 1706 ms). Trace:
I20260925 03:05:13.152560 27508 raft_consensus.cc:504] T e6b6ec010caf4689b8c90449764f5cf9 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:13.134346 24469 rpcz_store.cc:276] 0925 03:05:11.839387 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:05:12.612542 (+773155us) service_pool.cc:227] Handling call
0925 03:05:12.715319 (+102777us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:05:12.726737 (+ 11418us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:05:13.133501 (+406764us) spinlock_profiling.cc:243] Waited 76.1 ms on lock 0x7b5400090450. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764cfa09 00007f437b4112e9 00007f4374af7684 00007f4374af7612 00007f4374af7541 00007f4374af74bc 00007f4374af6772 00007f43748fa94c 00007f43748f9d86 00007f43748f972f
0925 03:05:13.133752 (+ 251us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":174941440}
I20260925 03:05:13.166962 24543 rpcz_store.cc:276] 0925 03:05:11.407854 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:05:11.407962 (+ 108us) service_pool.cc:227] Handling call
0925 03:05:11.408467 (+ 505us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:05:11.483262 (+ 74795us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:05:13.145473 (+1662211us) spinlock_profiling.cc:243] Waited 74.2 ms on lock 0x7b540010d454. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d0f51 00007f43764cf99e 00007f437b4112e9 00007f4374af7684 00007f4374af7612 00007f4374af7541 00007f4374af74bc 00007f4374af6772 00007f43748fa94c 00007f43748f9d86
0925 03:05:13.145758 (+ 285us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":170618240}
I20260925 03:05:13.172268 24544 rpcz_store.cc:276] 0925 03:05:09.934308 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:05:09.934446 (+ 138us) service_pool.cc:227] Handling call
0925 03:05:09.943813 (+ 9367us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:05:10.644331 (+700518us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:05:13.149177 (+2504846us) spinlock_profiling.cc:243] Waited 700 ms on lock 0x7b540004c9d4. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d0f51 00007f43764cf99e 00007f437b4112e9 00007f4374af7684 00007f4374af7612 00007f4374af7541 00007f4374af74bc 00007f4374af6772 00007f43748fa94c 00007f43748f9d86
0925 03:05:13.149537 (+ 360us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1609459072}
I20260925 03:05:13.141243 24548 raft_consensus.cc:3032] T 25102ba08e4e49388f6176badcd10a25 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 LEADER]: Stepping down as leader of term 6
I20260925 03:05:13.159809 27376 raft_consensus.cc:482] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260925 03:05:13.175220 24545 rpcz_store.cc:269] 0925 03:05:10.094968 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:05:10.095111 (+ 143us) service_pool.cc:227] Handling call
0925 03:05:13.150893 (+3055782us) spinlock_profiling.cc:243] Waited 829 ms on lock 0x7b54001d2c54. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d42ae 00007f437b411c27 00007f4374af9c04 00007f4374af9b92 00007f4374af9ac1 00007f4374af9a3c 00007f4374af8cf2 00007f43748fa94c 00007f43748f9d86 00007f43748f972f
0925 03:05:13.151121 (+ 228us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1906340096}
I20260925 03:05:13.179618 24548 raft_consensus.cc:729] T 25102ba08e4e49388f6176badcd10a25 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:05:13.180516 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74443a0a1dc34d88a2f42e3b722db405" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:13.182579 24546 raft_consensus.cc:1261] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Refusing update from remote peer 1bc1f2e62ab44af995165bd5116a9d43: Log matching property violated. Preceding OpId in replica: term: 6 index: 9. Preceding OpId from leader: term: 7 index: 9. (term mismatch)
I20260925 03:05:13.183005 24546 pending_rounds.cc:76] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5: Aborting all ops after (but not including) 8
I20260925 03:05:13.186054 24546 pending_rounds.cc:97] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5: Aborting uncommitted NO_OP operation due to leader change: 6.9
I20260925 03:05:13.186430 24546 raft_consensus.cc:2866] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260925 03:05:13.191027 27508 leader_election.cc:288] T e6b6ec010caf4689b8c90449764f5cf9 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:05:13.192144 27376 raft_consensus.cc:504] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:13.194221 24511 leader_election.cc:302] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:05:13.194881 27376 leader_election.cc:288] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:05:13.195572 27386 raft_consensus.cc:2781] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Leader election won for term 8
I20260925 03:05:13.196310 27386 raft_consensus.cc:686] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 LEADER]: Becoming Leader. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:05:13.181660 27346 raft_consensus.cc:2726] T 550118bc1a0641ce9803bdbf43f2069f P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260925 03:05:13.195585 27368 consensus_queue.cc:1046] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.015s
I20260925 03:05:13.213024 26829 leader_election.cc:288] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:05:13.213973 27368 raft_consensus.cc:3032] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 LEADER]: Stepping down as leader of term 7
I20260925 03:05:13.214533 27368 raft_consensus.cc:729] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:05:13.215816 27529 raft_consensus.cc:482] T 11bb46bce3564841b7674e6a693bd2e6 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:13.216710 27529 raft_consensus.cc:504] T 11bb46bce3564841b7674e6a693bd2e6 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:13.217739 27346 raft_consensus.cc:482] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:13.207808 27521 raft_consensus.cc:504] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:13.221496 27529 leader_election.cc:288] T 11bb46bce3564841b7674e6a693bd2e6 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:13.162145 24471 rpcz_store.cc:276] 0925 03:05:11.598258 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:05:11.618748 (+ 20490us) service_pool.cc:227] Handling call
0925 03:05:11.619086 (+ 338us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:05:11.736362 (+117276us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:05:11.736386 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:05:11.749109 (+ 12723us) log.cc:842] Serialized 26 byte log entry
0925 03:05:12.257765 (+508656us) raft_consensus.cc:1631] Marking committed up to 4
0925 03:05:12.759532 (+501767us) log.cc:842] Serialized 14 byte log entry
0925 03:05:12.768187 (+ 8655us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:05:13.136935 (+368748us) spinlock_profiling.cc:243] Waited 117 ms on lock 0x7b5400119ed4. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d0f51 00007f43764cf99e 00007f437b4112e9 00007f4374af7684 00007f4374af7612 00007f4374af7541 00007f4374af74bc 00007f4374af6772 00007f43748fa94c 00007f43748f9d86
0925 03:05:13.137007 (+ 72us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:05:13.137077 (+ 70us) raft_consensus.cc:1671] finished
0925 03:05:13.137180 (+ 103us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:05:13.137495 (+ 315us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8388,"prepare.queue_time_us":532403,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":44,"raft.queue_time_us":13385,"raft.run_cpu_time_us":337,"raft.run_wall_time_us":14347,"spinlock_wait_cycles":268875648,"thread_start_us":807521,"threads_started":2,"wal-append.queue_time_us":859484}
I20260925 03:05:13.203948 27386 consensus_queue.cc:237] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:13.228600 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e5819c15eac4544a57e5ee6c9d4f117" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:13.229704 24471 raft_consensus.cc:2454] T 8e5819c15eac4544a57e5ee6c9d4f117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 7.
W20260925 03:05:13.231312 27524 log.cc:924] Time spent T a7aa47c48404465b94a36fd819fe3c7a P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.115s user 0.001s sys 0.000s
W20260925 03:05:13.232858 25163 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 112962us
I20260925 03:05:13.240984 27328 raft_consensus.cc:482] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:13.215968 27368 consensus_queue.cc:260] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:13.243316 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:13.245441 27529 raft_consensus.cc:482] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:13.246444 27328 raft_consensus.cc:504] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:13.246940 27529 raft_consensus.cc:504] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:13.248052 24548 consensus_queue.cc:260] T 25102ba08e4e49388f6176badcd10a25 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:13.249480 24548 raft_consensus.cc:3037] T 25102ba08e4e49388f6176badcd10a25 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:05:13.249387 27254 raft_consensus.cc:2726] T c5aec38e66a947c681e6c2a72b05e6ca P 1bc1f2e62ab44af995165bd5116a9d43 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260925 03:05:13.250484 27328 leader_election.cc:288] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
W20260925 03:05:13.251731 27268 consensus_queue.cc:1159] T 25102ba08e4e49388f6176badcd10a25 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "eeab93abfdb141a895002ec9f312296c" responder_term: 7 status { last_received { term: 7 index: 4 } last_committed_idx: 3 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 6. Current term is 7. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260925 03:05:13.255191 27053 consensus_queue.cc:260] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:13.257021 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "420163a6cdce412ebc21c25fe3835638" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:13.218498 27346 raft_consensus.cc:504] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:05:13.264786 24585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.4:43979, user_credentials={real_user=slave}} blocked reactor thread for 176530us
I20260925 03:05:13.274391 24468 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1470) took 1418 ms. Trace:
I20260925 03:05:13.274900 24468 rpcz_store.cc:276] 0925 03:05:11.855862 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:05:12.975347 (+1119485us) service_pool.cc:227] Handling call
0925 03:05:12.976170 (+ 823us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:05:12.977241 (+ 1071us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:05:12.977295 (+ 54us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:05:12.978480 (+ 1185us) log.cc:842] Serialized 26 byte log entry
0925 03:05:13.076859 (+ 98379us) raft_consensus.cc:1631] Marking committed up to 3
0925 03:05:13.076910 (+ 51us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:05:13.076984 (+ 74us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:05:13.274174 (+197190us) raft_consensus.cc:1671] finished
0925 03:05:13.274195 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:05:13.274328 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":57521,"raft.run_cpu_time_us":302,"raft.run_wall_time_us":301,"thread_start_us":68681,"threads_started":1,"wal-append.queue_time_us":291162}
I20260925 03:05:13.276930 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9357d144aa49789faaf31fe5e6bf89" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:13.277236 25163 leader_election.cc:302] T e92f377a79354a6ebb4eec67303c13e3 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:05:13.278362 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceab6b0fccd54e909a64f66fe81a9a1f" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:13.278846 24588 leader_election.cc:302] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:05:13.280660 27533 raft_consensus.cc:2781] T e92f377a79354a6ebb4eec67303c13e3 P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260925 03:05:13.281294 27533 raft_consensus.cc:482] T e92f377a79354a6ebb4eec67303c13e3 P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:13.281808 27533 raft_consensus.cc:3037] T e92f377a79354a6ebb4eec67303c13e3 P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:05:13.281945 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcf6e6a422d4d198d4fbb0f7d3d845d" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
W20260925 03:05:13.283238 27436 log.cc:924] Time spent T 11bb46bce3564841b7674e6a693bd2e6 P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 2.506s user 0.014s sys 0.000s
I20260925 03:05:13.292007 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b030d84c670b48d5bff337ccc0c77626" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 11 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:13.298897 27536 raft_consensus.cc:2781] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260925 03:05:13.299489 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e92f377a79354a6ebb4eec67303c13e3" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:13.297811 24514 leader_election.cc:302] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:05:13.299424 27536 raft_consensus.cc:482] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:05:13.302939 27501 leader_election.cc:288] T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:05:13.303156 27536 raft_consensus.cc:3037] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Advancing to term 9
I20260925 03:05:13.298308 27529 leader_election.cc:288] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:13.328820 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:13.338517 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5332a5d329493991dff994e70ea92c" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:13.339326 24622 raft_consensus.cc:2454] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 8.
I20260925 03:05:13.342048 26800 raft_consensus.cc:482] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:13.342968 26800 raft_consensus.cc:504] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:05:13.346055 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 81918.2us
I20260925 03:05:13.349640 24614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48ca6ccd98194cf2af7b5278d15bf4ea" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:13.352295 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7691c6b475428c82d6181d693d26f3" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
W20260925 03:05:13.335609 24436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 47582.5us
I20260925 03:05:13.334489 24471 raft_consensus.cc:2454] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 6.
I20260925 03:05:13.346428 27279 raft_consensus.cc:2781] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260925 03:05:13.354897 24469 raft_consensus.cc:2454] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 9.
I20260925 03:05:13.330143 27521 leader_election.cc:288] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:05:13.356510 27279 raft_consensus.cc:482] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:05:13.361713 24436 leader_election.cc:302] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:05:13.358644 27513 raft_consensus.cc:482] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:13.364801 27505 raft_consensus.cc:482] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:13.364419 27513 raft_consensus.cc:504] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:13.365473 27505 raft_consensus.cc:504] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:13.332427 27493 leader_election.cc:288] T a4abe0ea757e4886be4945c5001e529a P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:13.366765 27440 raft_consensus.cc:482] T 43366a0a2f9d4344bf4e37ffd290ef8a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:13.367377 27440 raft_consensus.cc:504] T 43366a0a2f9d4344bf4e37ffd290ef8a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:13.367746 27505 leader_election.cc:288] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:13.382035 27499 raft_consensus.cc:2781] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [term 10 FOLLOWER]: Leader election won for term 10
I20260925 03:05:13.383049 24514 leader_election.cc:302] T 420163a6cdce412ebc21c25fe3835638 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:05:13.384475 27499 raft_consensus.cc:686] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [term 10 LEADER]: Becoming Leader. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:05:13.385774 27499 consensus_queue.cc:237] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:13.386468 24613 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:44570 (request call id 16) took 1957 ms. Trace:
I20260925 03:05:13.386844 24613 rpcz_store.cc:276] 0925 03:05:11.428842 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:05:11.429133 (+ 291us) service_pool.cc:227] Handling call
0925 03:05:13.386127 (+1956994us) spinlock_profiling.cc:243] Waited 63.1 ms on lock 0x7b54002b0f54. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764cabc3 00007f437b41481a 00007f4374b07d04 00007f4374b07c92 00007f4374b07bc1 00007f4374b07b3c 00007f4374b06df2 00007f43748fa94c 00007f43748f9d86 00007f43748f972f
0925 03:05:13.386263 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":145225472}
I20260925 03:05:13.369716 27279 raft_consensus.cc:3037] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Advancing to term 8
I20260925 03:05:13.388626 26800 leader_election.cc:288] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:13.364439 27451 leader_election.cc:288] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
W20260925 03:05:13.389560 24511 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 92526.9us
I20260925 03:05:13.391111 27268 consensus_queue.cc:1046] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260925 03:05:13.392151 27510 raft_consensus.cc:3032] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 LEADER]: Stepping down as leader of term 9
I20260925 03:05:13.392540 27510 raft_consensus.cc:729] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:05:13.389585 24546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33068 (request call id 202) took 2214 ms. Trace:
I20260925 03:05:13.393191 24546 rpcz_store.cc:276] 0925 03:05:11.175466 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:05:11.220568 (+ 45102us) service_pool.cc:227] Handling call
0925 03:05:13.166533 (+1945965us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:05:13.187121 (+ 20588us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:05:13.389371 (+202250us) spinlock_profiling.cc:243] Waited 1.93 s on lock 0x7b54001d2c50. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764cfa09 00007f437b4112e9 00007f4374af7684 00007f4374af7612 00007f4374af7541 00007f4374af74bc 00007f4374af6772 00007f43748fa94c 00007f43748f9d86 00007f43748f972f
0925 03:05:13.389526 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4449212672}
I20260925 03:05:13.393649 27510 consensus_queue.cc:260] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:13.378620 27440 leader_election.cc:288] T 43366a0a2f9d4344bf4e37ffd290ef8a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:13.383982 27505 consensus_queue.cc:1046] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.010s
I20260925 03:05:13.395197 24617 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38564 (request call id 1481) took 3633 ms. Trace:
I20260925 03:05:13.396030 27268 raft_consensus.cc:2781] T 420163a6cdce412ebc21c25fe3835638 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260925 03:05:13.397787 24437 leader_election.cc:302] T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:13.406692 24617 rpcz_store.cc:276] 0925 03:05:09.761467 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:05:09.768790 (+ 7323us) service_pool.cc:227] Handling call
0925 03:05:09.769086 (+ 296us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:05:10.661291 (+892205us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:05:13.394666 (+2733375us) spinlock_profiling.cc:243] Waited 885 ms on lock 0x7b54002b0554. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d0f51 00007f43764cf99e 00007f437b4112e9 00007f4374af7684 00007f4374af7612 00007f4374af7541 00007f4374af74bc 00007f4374af6772 00007f43748fa94c 00007f43748f9d86
0925 03:05:13.394996 (+ 330us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2035326848}
I20260925 03:05:13.408133 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442d4a29b1d743599f4b6b4871e26668" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:13.408962 24617 raft_consensus.cc:2454] T 442d4a29b1d743599f4b6b4871e26668 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 8.
I20260925 03:05:13.409361 27451 raft_consensus.cc:2726] T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260925 03:05:13.411012 24464 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1468) took 1558 ms. Trace:
I20260925 03:05:13.407001 27505 raft_consensus.cc:3032] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 LEADER]: Stepping down as leader of term 9
I20260925 03:05:13.402231 24620 raft_consensus.cc:1226] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c [term 10 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 9. Current term is 10. Ops: []
I20260925 03:05:13.402902 24614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48ca6ccd98194cf2af7b5278d15bf4ea" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:13.403729 25197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367277daf8d04778b3fa4ed94d71580d" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:05:13.404049 25196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:05:13.404909 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:13.405686 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367277daf8d04778b3fa4ed94d71580d" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:13.401285 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15861cd28e354598a3c6bfc4337c34bb" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:13.413575 27346 leader_election.cc:288] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:13.395117 27510 raft_consensus.cc:3037] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Advancing to term 10
I20260925 03:05:13.411396 24464 rpcz_store.cc:276] 0925 03:05:11.852102 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:05:12.715142 (+863040us) service_pool.cc:227] Handling call
0925 03:05:12.715386 (+ 244us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:05:12.978621 (+263235us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:05:12.978690 (+ 69us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:05:12.978757 (+ 67us) raft_consensus.cc:1631] Marking committed up to 5
0925 03:05:12.978831 (+ 74us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:05:13.410618 (+431787us) spinlock_profiling.cc:243] Waited 262 ms on lock 0x7b54001ab3d4. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d0f51 00007f43764cf99e 00007f437b4112e9 00007f4374af7684 00007f4374af7612 00007f4374af7541 00007f4374af74bc 00007f4374af6772 00007f43748fa94c 00007f43748f9d86
0925 03:05:13.410669 (+ 51us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:05:13.410835 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":603522304}
I20260925 03:05:13.407325 27268 raft_consensus.cc:482] T 420163a6cdce412ebc21c25fe3835638 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:13.429313 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:13.429565 27268 raft_consensus.cc:3037] T 420163a6cdce412ebc21c25fe3835638 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:05:13.430550 25197 raft_consensus.cc:2454] T 367277daf8d04778b3fa4ed94d71580d P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 6.
I20260925 03:05:13.431509 25196 raft_consensus.cc:2454] T 7e49248643df4131a1c42e74244e05c2 P 1bc1f2e62ab44af995165bd5116a9d43 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 8.
I20260925 03:05:13.433497 24436 leader_election.cc:302] T f22ddd7d1c444dc58f45a1fb0b3e02bc P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:05:13.433779 27403 raft_consensus.cc:482] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:13.434281 27529 consensus_queue.cc:1035] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260925 03:05:13.435562 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4abe0ea757e4886be4945c5001e529a" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:13.435995 24618 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38564 (request call id 1484) took 3673 ms. Trace:
I20260925 03:05:13.416019 27505 raft_consensus.cc:729] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:05:13.437188 24618 rpcz_store.cc:276] 0925 03:05:09.762260 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:05:09.763064 (+ 804us) service_pool.cc:227] Handling call
0925 03:05:09.763395 (+ 331us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:05:10.647854 (+884459us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:05:13.435506 (+2787652us) spinlock_profiling.cc:243] Waited 877 ms on lock 0x7b54000a5d54. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d0f51 00007f43764cf99e 00007f437b4112e9 00007f4374af7684 00007f4374af7612 00007f4374af7541 00007f4374af74bc 00007f4374af6772 00007f43748fa94c 00007f43748f9d86
0925 03:05:13.435816 (+ 310us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2017995520}
I20260925 03:05:13.437940 24614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcf6e6a422d4d198d4fbb0f7d3d845d" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:13.414319 27537 consensus_queue.cc:1035] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.025s
I20260925 03:05:13.434394 27403 raft_consensus.cc:504] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:13.440407 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5332a5d329493991dff994e70ea92c" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:13.447381 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f1fac028a144058b92331c0d9b44ebf" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:13.448033 24621 raft_consensus.cc:2454] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 8.
I20260925 03:05:13.446192 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f1fac028a144058b92331c0d9b44ebf" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:13.450309 27403 leader_election.cc:288] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:13.451717 24469 raft_consensus.cc:2454] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 6.
I20260925 03:05:13.461870 24514 leader_election.cc:302] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:05:13.459975 24614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c85f5d1d3f004cf5817bb594fd8e6469" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
W20260925 03:05:13.465555 25165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 352552us
I20260925 03:05:13.470069 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b5df303cd6a4d59aca5c18bbbff24ed" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:13.474210 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5332a5d329493991dff994e70ea92c" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:13.424944 27463 raft_consensus.cc:504] T 0be4e0aa984f4f2b81a58e382e434ad5 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:13.476070 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b8e0d260a884ecdb20f79de46c36b19" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:13.479686 24619 raft_consensus.cc:2454] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 9.
W20260925 03:05:13.488054 24616 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38570 (request call id 1664) took 3641 ms (client timeout 2141 ms). Trace:
I20260925 03:05:13.485829 27268 raft_consensus.cc:504] T 420163a6cdce412ebc21c25fe3835638 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:13.488299 24618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f68804eb5a244d87bb395d50683c6012" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
W20260925 03:05:13.488536 24616 rpcz_store.cc:269] 0925 03:05:09.841787 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:05:09.841971 (+ 184us) service_pool.cc:227] Handling call
0925 03:05:13.482596 (+3640625us) spinlock_profiling.cc:243] Waited 759 ms on lock 0x7b54000a6254. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d42ae 00007f437b411c27 00007f4374af9c04 00007f4374af9b92 00007f4374af9ac1 00007f4374af9a3c 00007f4374af8cf2 00007f43748fa94c 00007f43748f9d86 00007f43748f972f
0925 03:05:13.482775 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1745811328}
I20260925 03:05:13.489316 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "420163a6cdce412ebc21c25fe3835638" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:13.489491 27369 raft_consensus.cc:482] T 1414dcfe53f64a78ae413531eb8ec117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:13.475461 26800 raft_consensus.cc:2781] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260925 03:05:13.491125 24614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "100c113a9fd84cc8adf2ec7b33ea0093" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:13.488595 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b5df303cd6a4d59aca5c18bbbff24ed" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:13.486788 24468 raft_consensus.cc:2454] T 4e5332a5d329493991dff994e70ea92c P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 7.
I20260925 03:05:13.462668 27403 raft_consensus.cc:482] T 3d89396576884f7c8901dba54f783947 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:13.497570 27403 raft_consensus.cc:504] T 3d89396576884f7c8901dba54f783947 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:13.498113 27268 leader_election.cc:288] T 420163a6cdce412ebc21c25fe3835638 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 7 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
W20260925 03:05:13.500075 27466 log.cc:924] Time spent T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 1.979s user 0.004s sys 0.011s
I20260925 03:05:13.510247 27463 leader_election.cc:288] T 0be4e0aa984f4f2b81a58e382e434ad5 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 8 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:13.513752 27505 consensus_queue.cc:260] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:13.515462 27505 raft_consensus.cc:3037] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Advancing to term 10
I20260925 03:05:13.516500 27418 raft_consensus.cc:482] T 37a157996e944db2906110d87ecb0c54 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:05:13.517336 27418 raft_consensus.cc:504] T 37a157996e944db2906110d87ecb0c54 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:13.519579 27418 leader_election.cc:288] T 37a157996e944db2906110d87ecb0c54 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:13.490142 27369 raft_consensus.cc:504] T 1414dcfe53f64a78ae413531eb8ec117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:13.522013 27369 leader_election.cc:288] T 1414dcfe53f64a78ae413531eb8ec117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
W20260925 03:05:13.527475 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 152779us
I20260925 03:05:13.526823 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f68804eb5a244d87bb395d50683c6012" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:13.477795 27322 raft_consensus.cc:482] T 21c43af5e6844fdaacc8c625fa01fd8d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:13.541643 24616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e5819c15eac4544a57e5ee6c9d4f117" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:05:13.542191 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9357d144aa49789faaf31fe5e6bf89" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:13.544060 24467 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55560 (request call id 1521) took 2541 ms. Trace:
I20260925 03:05:13.544315 27346 raft_consensus.cc:482] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 1bc1f2e62ab44af995165bd5116a9d43)
I20260925 03:05:13.545604 24588 leader_election.cc:302] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:05:13.545159 24614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c85f5d1d3f004cf5817bb594fd8e6469" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
W20260925 03:05:13.541487 27526 log.cc:924] Time spent T 442d4a29b1d743599f4b6b4871e26668 P 869b0a8575b54d08ae2817fad5dd42d5: Append to log took a long time: real 0.066s user 0.002s sys 0.000s
I20260925 03:05:13.544600 24467 rpcz_store.cc:276] 0925 03:05:11.002081 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:05:11.002225 (+ 144us) service_pool.cc:227] Handling call
0925 03:05:11.220293 (+218068us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:05:11.447776 (+227483us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:05:13.543512 (+2095736us) spinlock_profiling.cc:243] Waited 206 ms on lock 0x7b54002f08d4. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d0f51 00007f43764cf99e 00007f437b4112e9 00007f4374af7684 00007f4374af7612 00007f4374af7541 00007f4374af74bc 00007f4374af6772 00007f43748fa94c 00007f43748f9d86
0925 03:05:13.543828 (+ 316us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":472976256}
I20260925 03:05:13.545994 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcf6e6a422d4d198d4fbb0f7d3d845d" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:13.547071 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e92f377a79354a6ebb4eec67303c13e3" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:13.558887 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e5819c15eac4544a57e5ee6c9d4f117" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:13.575367 24469 raft_consensus.cc:3037] T 8e5819c15eac4544a57e5ee6c9d4f117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Advancing to term 9
I20260925 03:05:13.562047 24548 raft_consensus.cc:1261] T 25102ba08e4e49388f6176badcd10a25 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 4. (term mismatch)
I20260925 03:05:13.494052 26800 raft_consensus.cc:482] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 1bc1f2e62ab44af995165bd5116a9d43)
I20260925 03:05:13.531075 27451 raft_consensus.cc:2781] T f22ddd7d1c444dc58f45a1fb0b3e02bc P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260925 03:05:13.544093 24436 leader_election.cc:302] T 4e5332a5d329493991dff994e70ea92c P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:05:13.578019 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c85f5d1d3f004cf5817bb594fd8e6469" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:13.576452 24548 pending_rounds.cc:76] T 25102ba08e4e49388f6176badcd10a25 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Aborting all ops after (but not including) 3
I20260925 03:05:13.529021 27463 raft_consensus.cc:482] T 5278a60f97da4e10aecd50a8d9a9a43c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:13.585902 23300 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260925 03:05:13.588217 24548 pending_rounds.cc:97] T 25102ba08e4e49388f6176badcd10a25 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Aborting uncommitted NO_OP operation due to leader change: 6.4
I20260925 03:05:13.588554 24613 raft_consensus.cc:1226] T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 7. Current term is 8. Ops: [7.6-7.6]
I20260925 03:05:13.581190 25197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15861cd28e354598a3c6bfc4337c34bb" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:05:13.588829 24548 raft_consensus.cc:2866] T 25102ba08e4e49388f6176badcd10a25 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260925 03:05:13.596345 24464 raft_consensus.cc:2454] T 7b8e0d260a884ecdb20f79de46c36b19 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 8.
I20260925 03:05:13.586860 26800 raft_consensus.cc:3037] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:05:13.586730 27451 raft_consensus.cc:482] T f22ddd7d1c444dc58f45a1fb0b3e02bc P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:05:13.603345 27385 raft_consensus.cc:2781] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260925 03:05:13.613324 27385 raft_consensus.cc:482] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:05:13.613744 27385 raft_consensus.cc:3037] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Advancing to term 9
I20260925 03:05:13.611821 24471 raft_consensus.cc:2454] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 7.
I20260925 03:05:13.614598 24547 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33068 (request call id 210) took 1001 ms. Trace:
I20260925 03:05:13.614740 25196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e92f377a79354a6ebb4eec67303c13e3" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
W20260925 03:05:13.613677 24436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 69837us
I20260925 03:05:13.615186 24547 rpcz_store.cc:276] 0925 03:05:12.608411 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:05:12.991634 (+383223us) service_pool.cc:227] Handling call
0925 03:05:12.992303 (+ 669us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:05:13.053265 (+ 60962us) raft_consensus.cc:1520] Early marking committed up to index 9
0925 03:05:13.165950 (+112685us) log.cc:842] Serialized 14 byte log entry
0925 03:05:13.609099 (+443149us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:05:13.609150 (+ 51us) raft_consensus.cc:1631] Marking committed up to 9
0925 03:05:13.609193 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:05:13.609254 (+ 61us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:05:13.609371 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":164447,"prepare.run_cpu_time_us":71,"prepare.run_wall_time_us":71,"raft.queue_time_us":10934,"thread_start_us":515219,"threads_started":2}
I20260925 03:05:13.621307 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceab6b0fccd54e909a64f66fe81a9a1f" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:13.621758 27427 consensus_queue.cc:1046] T 06f4a7eaf6534e4986c9fd35911038e4 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260925 03:05:13.619186 27508 raft_consensus.cc:2781] T 4e5332a5d329493991dff994e70ea92c P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260925 03:05:13.622691 27508 raft_consensus.cc:482] T 4e5332a5d329493991dff994e70ea92c P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:13.622900 27485 raft_consensus.cc:3032] T 06f4a7eaf6534e4986c9fd35911038e4 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 LEADER]: Stepping down as leader of term 8
I20260925 03:05:13.623440 27485 raft_consensus.cc:729] T 06f4a7eaf6534e4986c9fd35911038e4 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:05:13.614688 27451 raft_consensus.cc:3037] T f22ddd7d1c444dc58f45a1fb0b3e02bc P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Advancing to term 9
I20260925 03:05:13.624899 27485 consensus_queue.cc:260] T 06f4a7eaf6534e4986c9fd35911038e4 P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:05:13.625831 27539 log.cc:924] Time spent T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5: Append to log took a long time: real 0.148s user 0.000s sys 0.000s
W20260925 03:05:13.623026 27414 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 7 current term: 7 is higher.
I20260925 03:05:13.626578 27485 raft_consensus.cc:3037] T 06f4a7eaf6534e4986c9fd35911038e4 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Advancing to term 9
I20260925 03:05:13.626524 27346 raft_consensus.cc:504] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:13.629192 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9c33fd100ff44c1a3247319a324a044" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:05:13.630800 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b8e0d260a884ecdb20f79de46c36b19" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:13.632485 24548 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 1491) took 2190 ms. Trace:
I20260925 03:05:13.632898 24548 rpcz_store.cc:276] 0925 03:05:11.441953 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:05:12.649278 (+1207325us) service_pool.cc:227] Handling call
0925 03:05:12.649625 (+ 347us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:05:13.140977 (+491352us) spinlock_profiling.cc:243] Waited 1.71 ms on lock 0x7b54001c32a0. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f4376460730 00007f43764d2300 00007f43764d3371 00007f43764cfedf 00007f43764cf99e 00007f437b4112e9 00007f4374af7684 00007f4374af7612 00007f4374af7541 00007f4374af74bc
0925 03:05:13.616713 (+475736us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:05:13.632412 (+ 15699us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3927936}
I20260925 03:05:13.633888 27440 raft_consensus.cc:482] T 100c113a9fd84cc8adf2ec7b33ea0093 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:13.634702 27440 raft_consensus.cc:504] T 100c113a9fd84cc8adf2ec7b33ea0093 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:13.623577 24467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1414dcfe53f64a78ae413531eb8ec117" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:13.636551 24467 raft_consensus.cc:2454] T 1414dcfe53f64a78ae413531eb8ec117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 4.
I20260925 03:05:13.623420 24621 raft_consensus.cc:1226] T 74443a0a1dc34d88a2f42e3b722db405 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 6. Current term is 7. Ops: []
I20260925 03:05:13.629099 24618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43366a0a2f9d4344bf4e37ffd290ef8a" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:13.621780 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5332a5d329493991dff994e70ea92c" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:13.637334 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b8e0d260a884ecdb20f79de46c36b19" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:13.638409 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "420163a6cdce412ebc21c25fe3835638" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:13.650368 24464 raft_consensus.cc:3037] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:05:13.590891 27369 raft_consensus.cc:482] T b030d84c670b48d5bff337ccc0c77626 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:13.664889 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:13.665494 24468 raft_consensus.cc:2454] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 9.
I20260925 03:05:13.667356 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bb46bce3564841b7674e6a693bd2e6" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:13.667665 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35992316cb8d4c028333388eff02ecde" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:13.668228 27536 raft_consensus.cc:504] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:13.669086 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4abe0ea757e4886be4945c5001e529a" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:13.669463 24616 raft_consensus.cc:2379] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 9: Already voted for candidate eeab93abfdb141a895002ec9f312296c in this term.
I20260925 03:05:13.670159 24586 leader_election.cc:302] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:05:13.670878 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc2df40db08649c9a75e90000b878559" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:13.673359 27346 leader_election.cc:288] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:05:13.673949 27346 raft_consensus.cc:482] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:05:13.674526 27346 raft_consensus.cc:504] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:13.614003 27498 raft_consensus.cc:482] T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1bc1f2e62ab44af995165bd5116a9d43)
I20260925 03:05:13.676168 27386 raft_consensus.cc:482] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:13.676846 27386 raft_consensus.cc:504] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:13.679102 27386 leader_election.cc:288] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:13.685760 24469 raft_consensus.cc:2454] T 8e5819c15eac4544a57e5ee6c9d4f117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 9.
I20260925 03:05:13.670413 27536 leader_election.cc:288] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 9 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:05:13.669751 24619 raft_consensus.cc:2454] T a4abe0ea757e4886be4945c5001e529a P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 3.
I20260925 03:05:13.612916 27463 raft_consensus.cc:504] T 5278a60f97da4e10aecd50a8d9a9a43c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:13.649767 27508 raft_consensus.cc:3037] T 4e5332a5d329493991dff994e70ea92c P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Advancing to term 8
I20260925 03:05:13.687273 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37a157996e944db2906110d87ecb0c54" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:13.687655 24467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06f4a7eaf6534e4986c9fd35911038e4" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:13.694769 25197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ac466793d094325ace224708b2158ed" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:05:13.695371 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e8af2641e124f9589c5ccfdac785f63" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 9 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:13.655449 27427 raft_consensus.cc:482] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:13.699371 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367277daf8d04778b3fa4ed94d71580d" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:13.686537 27346 leader_election.cc:288] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
W20260925 03:05:13.701952 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 124740us
I20260925 03:05:13.684147 27279 raft_consensus.cc:504] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:13.703868 24542 raft_consensus.cc:2454] T 367277daf8d04778b3fa4ed94d71580d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 6.
I20260925 03:05:13.704316 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b5df303cd6a4d59aca5c18bbbff24ed" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:13.704746 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b627ecb4a5384ef392aa695e06599c39" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:13.616364 27322 raft_consensus.cc:504] T 21c43af5e6844fdaacc8c625fa01fd8d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:13.705986 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9357d144aa49789faaf31fe5e6bf89" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:13.709069 25199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4abe0ea757e4886be4945c5001e529a" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:05:13.710417 27463 leader_election.cc:288] T 5278a60f97da4e10aecd50a8d9a9a43c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:13.695974 27369 raft_consensus.cc:504] T b030d84c670b48d5bff337ccc0c77626 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:13.676807 27498 raft_consensus.cc:504] T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:13.711594 27368 raft_consensus.cc:3037] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Advancing to term 8
I20260925 03:05:13.712085 24465 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1464) took 1863 ms. Trace:
I20260925 03:05:13.712235 24619 raft_consensus.cc:1261] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260925 03:05:13.713150 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceab6b0fccd54e909a64f66fe81a9a1f" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:05:13.713714 27427 raft_consensus.cc:504] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:13.721372 24546 raft_consensus.cc:1226] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Rejecting Update request from peer 1bc1f2e62ab44af995165bd5116a9d43 for earlier term 7. Current term is 8. Ops: [7.9-7.9]
I20260925 03:05:13.715154 25199 raft_consensus.cc:2454] T a4abe0ea757e4886be4945c5001e529a P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 3.
I20260925 03:05:13.718267 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e5819c15eac4544a57e5ee6c9d4f117" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:13.720671 24545 raft_consensus.cc:3037] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Advancing to term 9
I20260925 03:05:13.716673 27536 raft_consensus.cc:2726] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260925 03:05:13.721570 24618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5278a60f97da4e10aecd50a8d9a9a43c" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:13.726397 24471 raft_consensus.cc:2379] T 8e5819c15eac4544a57e5ee6c9d4f117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c in current term 9: Already voted for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in this term.
I20260925 03:05:13.720371 24465 rpcz_store.cc:276] 0925 03:05:11.848971 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:05:12.650697 (+801726us) service_pool.cc:227] Handling call
0925 03:05:12.670465 (+ 19768us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:05:12.725579 (+ 55114us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:05:12.725608 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:05:12.725651 (+ 43us) raft_consensus.cc:1631] Marking committed up to 3
0925 03:05:12.725680 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:05:13.711672 (+985992us) spinlock_profiling.cc:243] Waited 54.9 ms on lock 0x7b54003eadd4. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d0f51 00007f43764cf99e 00007f437b4112e9 00007f4374af7684 00007f4374af7612 00007f4374af7541 00007f4374af74bc 00007f4374af6772 00007f43748fa94c 00007f43748f9d86
0925 03:05:13.711733 (+ 61us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:05:13.711902 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":126291840}
I20260925 03:05:13.721611 27440 leader_election.cc:288] T 100c113a9fd84cc8adf2ec7b33ea0093 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:13.727787 24621 raft_consensus.cc:2454] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 8.
I20260925 03:05:13.733505 25199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4ebd2b043024a548005c8b6b00b0449" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:05:13.734346 25191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15861cd28e354598a3c6bfc4337c34bb" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:05:13.736563 24543 raft_consensus.cc:1261] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260925 03:05:13.737886 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5332a5d329493991dff994e70ea92c" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:13.738648 24543 raft_consensus.cc:2379] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 in current term 8: Already voted for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in this term.
I20260925 03:05:13.740033 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60d45b7d467d4e1ab92367f6d7d52b15" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:05:13.741094 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48ca6ccd98194cf2af7b5278d15bf4ea" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:13.741289 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b627ecb4a5384ef392aa695e06599c39" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:13.742290 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c85f5d1d3f004cf5817bb594fd8e6469" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:13.742493 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcf6e6a422d4d198d4fbb0f7d3d845d" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:13.742959 24539 raft_consensus.cc:2454] T c85f5d1d3f004cf5817bb594fd8e6469 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 4.
I20260925 03:05:13.743191 24541 raft_consensus.cc:2454] T edcf6e6a422d4d198d4fbb0f7d3d845d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 5.
I20260925 03:05:13.744161 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9c33fd100ff44c1a3247319a324a044" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:05:13.752231 24469 raft_consensus.cc:2396] T b627ecb4a5384ef392aa695e06599c39 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c for term 9 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20260925 03:05:13.744540 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "293ef69caf514ac98510a68495f54eda" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 8 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:13.751365 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "100c113a9fd84cc8adf2ec7b33ea0093" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:13.744669 24541 raft_consensus.cc:3037] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Advancing to term 9
I20260925 03:05:13.753077 24547 raft_consensus.cc:2454] T ac9357d144aa49789faaf31fe5e6bf89 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 2.
W20260925 03:05:13.760952 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 108038us
I20260925 03:05:13.761986 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ac285175464fbda67211d9574db415" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:13.764032 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4ebd2b043024a548005c8b6b00b0449" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:13.749627 27322 leader_election.cc:288] T 21c43af5e6844fdaacc8c625fa01fd8d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:13.767969 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43366a0a2f9d4344bf4e37ffd290ef8a" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:13.769713 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f68804eb5a244d87bb395d50683c6012" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:13.770569 24539 raft_consensus.cc:2454] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 7.
I20260925 03:05:13.770676 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15861cd28e354598a3c6bfc4337c34bb" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:13.751652 27463 raft_consensus.cc:482] T 550118bc1a0641ce9803bdbf43f2069f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:13.772722 24514 leader_election.cc:302] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:05:13.773245 24618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abd5b8dc1b7e49e3aa9af900d73ff0e7" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:13.769284 24465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7691c6b475428c82d6181d693d26f3" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:13.767215 27498 leader_election.cc:288] T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:05:13.775553 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f22ddd7d1c444dc58f45a1fb0b3e02bc" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:13.775209 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be4e0aa984f4f2b81a58e382e434ad5" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:13.775812 24468 raft_consensus.cc:2454] T 100c113a9fd84cc8adf2ec7b33ea0093 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 3.
I20260925 03:05:13.775100 24621 raft_consensus.cc:2454] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 8.
I20260925 03:05:13.765834 27403 leader_election.cc:288] T 3d89396576884f7c8901dba54f783947 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:13.779953 24470 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1434) took 3149 ms. Trace:
I20260925 03:05:13.780319 24470 rpcz_store.cc:276] 0925 03:05:10.630876 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:05:10.631556 (+ 680us) service_pool.cc:227] Handling call
0925 03:05:10.631910 (+ 354us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:05:11.419050 (+787140us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:05:13.779533 (+2360483us) spinlock_profiling.cc:243] Waited 786 ms on lock 0x7b54003f7d54. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d0f51 00007f43764cf99e 00007f437b4112e9 00007f4374af7684 00007f4374af7612 00007f4374af7541 00007f4374af74bc 00007f4374af6772 00007f43748fa94c 00007f43748f9d86
0925 03:05:13.779793 (+ 260us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1808935168}
W20260925 03:05:13.781073 24473 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55560 (request call id 1538) took 1939 ms (client timeout 1667 ms). Trace:
I20260925 03:05:13.781486 24466 raft_consensus.cc:1261] T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260925 03:05:13.782181 24470 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38920 (request call id 222) took 1925 ms. Trace:
I20260925 03:05:13.781105 24619 raft_consensus.cc:1261] T 442d4a29b1d743599f4b6b4871e26668 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260925 03:05:13.783151 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e54ca1757ee4e42956a9cebbeca4f62" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:13.782552 24470 rpcz_store.cc:276] 0925 03:05:11.856330 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:05:13.781275 (+1924945us) service_pool.cc:227] Handling call
0925 03:05:13.781609 (+ 334us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:05:13.781855 (+ 246us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:05:13.781879 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:05:13.781937 (+ 58us) raft_consensus.cc:1631] Marking committed up to 6
0925 03:05:13.781962 (+ 25us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:05:13.782016 (+ 54us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:05:13.782112 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:05:13.788120 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b5df303cd6a4d59aca5c18bbbff24ed" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:13.789523 24471 raft_consensus.cc:3037] T 0be4e0aa984f4f2b81a58e382e434ad5 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Advancing to term 8
W20260925 03:05:13.781499 24473 rpcz_store.cc:269] 0925 03:05:11.841126 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:05:11.844334 (+ 3208us) service_pool.cc:227] Handling call
0925 03:05:13.780675 (+1936341us) spinlock_profiling.cc:243] Waited 682 ms on lock 0x7b540019f354. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d42ae 00007f437b411c27 00007f4374af9c04 00007f4374af9b92 00007f4374af9ac1 00007f4374af9a3c 00007f4374af8cf2 00007f43748fa94c 00007f43748f9d86 00007f43748f972f
0925 03:05:13.780867 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1569254272}
I20260925 03:05:13.792363 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "100c113a9fd84cc8adf2ec7b33ea0093" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:13.793108 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7691c6b475428c82d6181d693d26f3" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:13.793592 24547 raft_consensus.cc:2454] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 8.
I20260925 03:05:13.792933 24543 raft_consensus.cc:2454] T 100c113a9fd84cc8adf2ec7b33ea0093 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 3.
I20260925 03:05:13.794821 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35992316cb8d4c028333388eff02ecde" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:13.796674 25197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6b6ec010caf4689b8c90449764f5cf9" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:05:13.798399 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b8e0d260a884ecdb20f79de46c36b19" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:13.800438 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e8af2641e124f9589c5ccfdac785f63" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 9 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:13.803978 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e725b82985746b39c0cc8f671c95353" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:13.789516 27498 raft_consensus.cc:482] T 3d89396576884f7c8901dba54f783947 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:13.800659 24465 raft_consensus.cc:2454] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 8.
I20260925 03:05:13.806533 24543 raft_consensus.cc:1261] T b030d84c670b48d5bff337ccc0c77626 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 10 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 10 index: 5. (index mismatch)
I20260925 03:05:13.804553 24514 leader_election.cc:302] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:05:13.808775 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc2df40db08649c9a75e90000b878559" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:13.805497 27479 raft_consensus.cc:2781] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260925 03:05:13.809650 27479 raft_consensus.cc:482] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:05:13.810031 27479 raft_consensus.cc:3037] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Advancing to term 9
I20260925 03:05:13.801519 27379 consensus_queue.cc:1046] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.009s
I20260925 03:05:13.810280 24473 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38920 (request call id 223) took 1926 ms. Trace:
I20260925 03:05:13.805430 27022 consensus_queue.cc:1035] T 11bb46bce3564841b7674e6a693bd2e6 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.020s
I20260925 03:05:13.810618 27403 consensus_queue.cc:1046] T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260925 03:05:13.811539 27403 raft_consensus.cc:3032] T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 LEADER]: Stepping down as leader of term 6
I20260925 03:05:13.811882 27403 raft_consensus.cc:729] T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:05:13.812274 27299 raft_consensus.cc:3037] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Advancing to term 8
I20260925 03:05:13.813163 27369 leader_election.cc:288] T b030d84c670b48d5bff337ccc0c77626 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:13.801882 25191 raft_consensus.cc:1226] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43 [term 7 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 6. Current term is 7. Ops: [6.8-6.8]
I20260925 03:05:13.826608 24464 raft_consensus.cc:2454] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 7.
I20260925 03:05:13.799401 27499 raft_consensus.cc:482] T a7aa47c48404465b94a36fd819fe3c7a P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:13.810642 24473 rpcz_store.cc:276] 0925 03:05:11.884089 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:05:13.800850 (+1916761us) service_pool.cc:227] Handling call
0925 03:05:13.809726 (+ 8876us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:05:13.809942 (+ 216us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:05:13.809968 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:05:13.810029 (+ 61us) raft_consensus.cc:1631] Marking committed up to 6
0925 03:05:13.810062 (+ 33us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:05:13.810110 (+ 48us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:05:13.810210 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:05:13.820832 24465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:13.794552 27463 raft_consensus.cc:504] T 550118bc1a0641ce9803bdbf43f2069f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:13.827196 24619 raft_consensus.cc:2379] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 in current term 8: Already voted for candidate eeab93abfdb141a895002ec9f312296c in this term.
I20260925 03:05:13.819382 24468 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1471) took 1959 ms. Trace:
I20260925 03:05:13.826771 25197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e54ca1757ee4e42956a9cebbeca4f62" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:05:13.829279 27481 raft_consensus.cc:2781] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260925 03:05:13.825135 24514 leader_election.cc:302] T 1414dcfe53f64a78ae413531eb8ec117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:05:13.830983 24468 rpcz_store.cc:276] 0925 03:05:11.856153 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:05:13.785288 (+1929135us) service_pool.cc:227] Handling call
0925 03:05:13.804833 (+ 19545us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:05:13.814777 (+ 9944us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:05:13.814802 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:05:13.814842 (+ 40us) raft_consensus.cc:1631] Marking committed up to 5
0925 03:05:13.814867 (+ 25us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:05:13.814923 (+ 56us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:05:13.815032 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1153792}
I20260925 03:05:13.831512 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6b6ec010caf4689b8c90449764f5cf9" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:13.808607 27427 leader_election.cc:288] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:13.834499 24469 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55560 (request call id 1542) took 1476 ms. Trace:
W20260925 03:05:13.830272 27443 log.cc:924] Time spent T c88cbd39e01f44628a094af516be2833 P 869b0a8575b54d08ae2817fad5dd42d5: Append to log took a long time: real 3.054s user 0.003s sys 0.011s
I20260925 03:05:13.837325 24544 raft_consensus.cc:1261] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260925 03:05:13.837921 24472 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55560 (request call id 1517) took 3834 ms. Trace:
I20260925 03:05:13.838322 24472 rpcz_store.cc:276] 0925 03:05:10.003041 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:05:10.052114 (+ 49073us) service_pool.cc:227] Handling call
0925 03:05:10.052401 (+ 287us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:05:10.674125 (+621724us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:05:10.674157 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:05:10.674614 (+ 457us) log.cc:842] Serialized 26 byte log entry
0925 03:05:10.767044 (+ 92430us) raft_consensus.cc:1631] Marking committed up to 3
0925 03:05:10.767083 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:05:10.767156 (+ 73us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:05:13.829430 (+3062274us) spinlock_profiling.cc:243] Waited 4.27 ms on lock 0x7b44000232cc. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f43733ecf11 00007f43734baa95 00007f43734bb4b2 00007f4376163511 00007f437615ee1a 00007f4376152bee 00007f4376151f36 00007f4376151b58 00007f437615b551 00007f437615b509 00007f437615b499
0925 03:05:13.837466 (+ 8036us) raft_consensus.cc:1671] finished
0925 03:05:13.837486 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:05:13.837744 (+ 258us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":97957,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":6260,"spinlock_wait_cycles":9817984,"thread_start_us":692446,"threads_started":2,"wal-append.queue_time_us":101631}
I20260925 03:05:13.842237 24469 rpcz_store.cc:276] 0925 03:05:12.357638 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:05:13.803180 (+1445542us) service_pool.cc:227] Handling call
0925 03:05:13.828330 (+ 25150us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:05:13.834069 (+ 5739us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:05:13.834092 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:05:13.834232 (+ 140us) raft_consensus.cc:1631] Marking committed up to 5
0925 03:05:13.834261 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:05:13.834306 (+ 45us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:05:13.834446 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:05:13.838426 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bb46bce3564841b7674e6a693bd2e6" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:13.843773 24470 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1472) took 1987 ms. Trace:
I20260925 03:05:13.844137 24470 rpcz_store.cc:276] 0925 03:05:11.856425 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:05:13.787909 (+1931484us) service_pool.cc:227] Handling call
0925 03:05:13.803206 (+ 15297us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:05:13.830663 (+ 27457us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:05:13.830690 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:05:13.843506 (+ 12816us) raft_consensus.cc:1631] Marking committed up to 5
0925 03:05:13.843564 (+ 58us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:05:13.843618 (+ 54us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:05:13.843719 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:05:13.845695 24465 raft_consensus.cc:2454] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 8.
I20260925 03:05:13.846164 24514 leader_election.cc:302] T 8e5819c15eac4544a57e5ee6c9d4f117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:05:13.850276 24468 raft_consensus.cc:1261] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260925 03:05:13.833016 27481 raft_consensus.cc:482] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:13.853994 24544 raft_consensus.cc:2454] T 11bb46bce3564841b7674e6a693bd2e6 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 5.
I20260925 03:05:13.835465 27369 raft_consensus.cc:2781] T 1414dcfe53f64a78ae413531eb8ec117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260925 03:05:13.858574 24473 raft_consensus.cc:1261] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260925 03:05:13.835340 27279 leader_election.cc:288] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 8 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:05:13.862501 25197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1e85248d3b04b979dd080caeeb02fdb" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:05:13.864593 24545 raft_consensus.cc:2396] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 for term 9 because replica has last-logged OpId of term: 8 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 3.
I20260925 03:05:13.865567 24465 raft_consensus.cc:1226] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 6. Current term is 7. Ops: []
I20260925 03:05:13.866675 27460 raft_consensus.cc:2781] T 8e5819c15eac4544a57e5ee6c9d4f117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Leader election won for term 9
I20260925 03:05:13.871304 27267 raft_consensus.cc:3032] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 LEADER]: Stepping down as leader of term 6
I20260925 03:05:13.871853 27267 raft_consensus.cc:729] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:05:13.862263 27403 consensus_queue.cc:260] T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:13.875413 27403 raft_consensus.cc:3037] T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:05:13.880462 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5278a60f97da4e10aecd50a8d9a9a43c" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:13.826581 27498 raft_consensus.cc:504] T 3d89396576884f7c8901dba54f783947 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:13.880952 27460 raft_consensus.cc:686] T 8e5819c15eac4544a57e5ee6c9d4f117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 LEADER]: Becoming Leader. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:05:13.862319 27369 raft_consensus.cc:482] T 1414dcfe53f64a78ae413531eb8ec117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:13.881613 24545 raft_consensus.cc:2454] T 5278a60f97da4e10aecd50a8d9a9a43c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 6.
I20260925 03:05:13.887269 23300 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260925 03:05:13.862854 27481 raft_consensus.cc:3037] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Advancing to term 8
I20260925 03:05:13.852856 27499 raft_consensus.cc:504] T a7aa47c48404465b94a36fd819fe3c7a P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:05:13.940492 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 87973.4us
I20260925 03:05:13.949195 25162 leader_election.cc:302] T 29637f5a4a504ce6ba29033ccd0cd716 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:05:13.949607 24618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c23337a7cb74184a0e57b7095cb6a11" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:13.959883 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e725b82985746b39c0cc8f671c95353" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:05:13.960860 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e8af2641e124f9589c5ccfdac785f63" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:13.965312 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74443a0a1dc34d88a2f42e3b722db405" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:05:13.965864 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7993f9c3e2a44f408bbf5d1e395e2306" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:13.974239 24539 raft_consensus.cc:1261] T 442d4a29b1d743599f4b6b4871e26668 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
W20260925 03:05:13.941365 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 63541.7us
I20260925 03:05:13.970868 25197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 10 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:05:13.971159 25165 leader_election.cc:302] T 7e49248643df4131a1c42e74244e05c2 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:05:13.954573 24541 raft_consensus.cc:2396] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 for term 9 because replica has last-logged OpId of term: 8 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260925 03:05:13.914476 24466 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1463) took 2065 ms. Trace:
I20260925 03:05:13.981854 25162 leader_election.cc:302] T a4abe0ea757e4886be4945c5001e529a P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, eeab93abfdb141a895002ec9f312296c; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:05:13.978030 24514 leader_election.cc:302] T 100c113a9fd84cc8adf2ec7b33ea0093 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:05:13.942864 27053 raft_consensus.cc:3037] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Advancing to term 9
W20260925 03:05:13.947185 27533 consensus_meta.cc:287] T e92f377a79354a6ebb4eec67303c13e3 P 1bc1f2e62ab44af995165bd5116a9d43: Time spent flushing consensus metadata: real 0.664s user 0.006s sys 0.006s
I20260925 03:05:13.974215 27479 raft_consensus.cc:504] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:14.002823 27533 raft_consensus.cc:504] T e92f377a79354a6ebb4eec67303c13e3 P 1bc1f2e62ab44af995165bd5116a9d43 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:14.004317 24544 raft_consensus.cc:2454] T 7993f9c3e2a44f408bbf5d1e395e2306 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 5.
I20260925 03:05:14.002203 24473 raft_consensus.cc:1226] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 8. Current term is 9. Ops: [8.4-8.4]
I20260925 03:05:14.005194 24467 raft_consensus.cc:2454] T 06f4a7eaf6534e4986c9fd35911038e4 P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 9.
I20260925 03:05:13.969063 24464 raft_consensus.cc:1226] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 LEADER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 8. Current term is 9. Ops: [8.6-8.6]
I20260925 03:05:13.948074 24616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1e85248d3b04b979dd080caeeb02fdb" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:13.998379 24466 rpcz_store.cc:276] 0925 03:05:11.848661 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:05:12.626204 (+777543us) service_pool.cc:227] Handling call
0925 03:05:13.780926 (+1154722us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:05:13.781882 (+ 956us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:05:13.914293 (+132411us) spinlock_profiling.cc:243] Waited 1.15 s on lock 0x7b540019f350. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764cfa09 00007f437b4112e9 00007f4374af7684 00007f4374af7612 00007f4374af7541 00007f4374af74bc 00007f4374af6772 00007f43748fa94c 00007f43748f9d86 00007f43748f972f
0925 03:05:13.914416 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2642867968}
I20260925 03:05:14.009380 24542 raft_consensus.cc:2379] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 in current term 9: Already voted for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in this term.
I20260925 03:05:14.009840 24472 raft_consensus.cc:1261] T 9e8af2641e124f9589c5ccfdac785f63 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260925 03:05:14.010768 25196 raft_consensus.cc:2379] T e92f377a79354a6ebb4eec67303c13e3 P 1bc1f2e62ab44af995165bd5116a9d43 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 7: Already voted for candidate 1bc1f2e62ab44af995165bd5116a9d43 in this term.
I20260925 03:05:14.011409 24543 raft_consensus.cc:1261] T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 5. (term mismatch)
W20260925 03:05:13.917076 27547 log.cc:924] Time spent T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.239s user 0.004s sys 0.000s
I20260925 03:05:14.014878 24464 raft_consensus.cc:1261] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260925 03:05:13.907476 27268 raft_consensus.cc:482] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260925 03:05:13.972390 27552 log.cc:924] Time spent T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5: Append to log took a long time: real 0.173s user 0.000s sys 0.000s
I20260925 03:05:13.965471 27427 raft_consensus.cc:482] T 25102ba08e4e49388f6176badcd10a25 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:14.029088 27427 raft_consensus.cc:504] T 25102ba08e4e49388f6176badcd10a25 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:13.930732 27463 leader_election.cc:288] T 550118bc1a0641ce9803bdbf43f2069f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:14.030472 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367277daf8d04778b3fa4ed94d71580d" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:14.011793 24543 pending_rounds.cc:76] T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Aborting all ops after (but not including) 4
I20260925 03:05:14.032276 24543 pending_rounds.cc:97] T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Aborting uncommitted NO_OP operation due to leader change: 8.5
I20260925 03:05:14.032580 24543 raft_consensus.cc:2866] T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260925 03:05:13.947026 27369 raft_consensus.cc:3037] T 1414dcfe53f64a78ae413531eb8ec117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:05:14.014773 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9357d144aa49789faaf31fe5e6bf89" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:13.945140 27498 leader_election.cc:288] T 3d89396576884f7c8901dba54f783947 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:14.020267 24540 raft_consensus.cc:2396] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 for term 10 because replica has last-logged OpId of term: 9 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 4.
I20260925 03:05:14.039122 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceab6b0fccd54e909a64f66fe81a9a1f" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:05:14.040068 24545 raft_consensus.cc:2396] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 for term 9 because replica has last-logged OpId of term: 8 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 3.
I20260925 03:05:13.932183 27022 raft_consensus.cc:3032] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [term 8 LEADER]: Stepping down as leader of term 8
I20260925 03:05:14.031160 24539 raft_consensus.cc:2454] T 367277daf8d04778b3fa4ed94d71580d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 6.
I20260925 03:05:13.946884 23300 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260925 03:05:14.004489 27479 leader_election.cc:288] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 9 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:14.020051 27536 consensus_queue.cc:1035] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.005s
I20260925 03:05:14.024799 27534 raft_consensus.cc:2781] T 29637f5a4a504ce6ba29033ccd0cd716 P 1bc1f2e62ab44af995165bd5116a9d43 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260925 03:05:14.025075 27434 raft_consensus.cc:2781] T a4abe0ea757e4886be4945c5001e529a P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260925 03:05:14.012336 27510 raft_consensus.cc:2781] T 100c113a9fd84cc8adf2ec7b33ea0093 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260925 03:05:14.040552 24514 leader_election.cc:302] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:05:13.906071 27513 leader_election.cc:288] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:05:14.050199 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ac285175464fbda67211d9574db415" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:14.050571 24585 leader_election.cc:302] T a4abe0ea757e4886be4945c5001e529a P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:05:14.058533 24436 leader_election.cc:302] T 9e8af2641e124f9589c5ccfdac785f63 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:05:14.085947 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ac466793d094325ace224708b2158ed" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:13.929374 27267 consensus_queue.cc:260] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:13.893196 27385 raft_consensus.cc:504] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:14.062368 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5332a5d329493991dff994e70ea92c" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:14.103720 24543 raft_consensus.cc:2454] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 8.
I20260925 03:05:14.062314 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abd5b8dc1b7e49e3aa9af900d73ff0e7" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:14.085295 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4abe0ea757e4886be4945c5001e529a" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:13.888669 26800 raft_consensus.cc:504] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
W20260925 03:05:14.105772 24585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.4:43979, user_credentials={real_user=slave}} blocked reactor thread for 55271.3us
I20260925 03:05:14.046324 27022 raft_consensus.cc:729] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [term 8 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:05:14.086277 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e5819c15eac4544a57e5ee6c9d4f117" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:14.108433 24436 leader_election.cc:302] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:05:14.009949 27508 raft_consensus.cc:504] T 4e5332a5d329493991dff994e70ea92c P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:14.107908 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e92f377a79354a6ebb4eec67303c13e3" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:05:14.016175 27451 raft_consensus.cc:504] T f22ddd7d1c444dc58f45a1fb0b3e02bc P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:13.937456 24367 catalog_manager.cc:5495] T 367277daf8d04778b3fa4ed94d71580d P 1bc1f2e62ab44af995165bd5116a9d43 reported cstate change: term changed from 5 to 6, leader changed from 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) to eeab93abfdb141a895002ec9f312296c (127.22.193.3). New cstate: current_term: 6 leader_uuid: "eeab93abfdb141a895002ec9f312296c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:05:14.043913 27533 leader_election.cc:288] T e92f377a79354a6ebb4eec67303c13e3 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 7 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:14.050848 27339 raft_consensus.cc:2781] T 7e49248643df4131a1c42e74244e05c2 P 1bc1f2e62ab44af995165bd5116a9d43 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260925 03:05:13.945497 27460 consensus_queue.cc:237] T 8e5819c15eac4544a57e5ee6c9d4f117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:14.116216 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f1fac028a144058b92331c0d9b44ebf" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:14.119712 24622 raft_consensus.cc:2379] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 9: Already voted for candidate eeab93abfdb141a895002ec9f312296c in this term.
I20260925 03:05:14.116185 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e5819c15eac4544a57e5ee6c9d4f117" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:14.088541 27254 raft_consensus.cc:482] T 04e068a0f80941dc9138cee063f53bab P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:14.136418 27254 raft_consensus.cc:504] T 04e068a0f80941dc9138cee063f53bab P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:14.059653 27534 raft_consensus.cc:482] T 29637f5a4a504ce6ba29033ccd0cd716 P 1bc1f2e62ab44af995165bd5116a9d43 [term 9 FOLLOWER]: Starting leader election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:05:14.126827 24614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d89396576884f7c8901dba54f783947" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:14.120385 27339 raft_consensus.cc:482] T 7e49248643df4131a1c42e74244e05c2 P 1bc1f2e62ab44af995165bd5116a9d43 [term 8 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:14.138302 27339 raft_consensus.cc:3037] T 7e49248643df4131a1c42e74244e05c2 P 1bc1f2e62ab44af995165bd5116a9d43 [term 8 FOLLOWER]: Advancing to term 9
I20260925 03:05:14.127630 24616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 10 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:14.139246 24616 raft_consensus.cc:2396] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 for term 10 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20260925 03:05:14.109089 27513 raft_consensus.cc:2781] T 9e8af2641e124f9589c5ccfdac785f63 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260925 03:05:14.140080 27513 raft_consensus.cc:482] T 9e8af2641e124f9589c5ccfdac785f63 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:05:14.041109 27268 raft_consensus.cc:504] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:14.140386 24614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e068a0f80941dc9138cee063f53bab" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:14.141671 24540 raft_consensus.cc:2396] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 for term 9 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260925 03:05:14.143467 27268 leader_election.cc:288] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:14.116533 27241 consensus_queue.cc:1046] T 420163a6cdce412ebc21c25fe3835638 P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.019s
I20260925 03:05:14.145258 27241 raft_consensus.cc:3032] T 420163a6cdce412ebc21c25fe3835638 P eeab93abfdb141a895002ec9f312296c [term 6 LEADER]: Stepping down as leader of term 6
I20260925 03:05:14.145591 27241 raft_consensus.cc:729] T 420163a6cdce412ebc21c25fe3835638 P eeab93abfdb141a895002ec9f312296c [term 6 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:05:14.146476 27241 consensus_queue.cc:260] T 420163a6cdce412ebc21c25fe3835638 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:14.147701 27241 raft_consensus.cc:3037] T 420163a6cdce412ebc21c25fe3835638 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:05:14.150290 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e8af2641e124f9589c5ccfdac785f63" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:14.151767 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b030d84c670b48d5bff337ccc0c77626" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 11 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:14.152386 24540 raft_consensus.cc:2454] T b030d84c670b48d5bff337ccc0c77626 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 10.
I20260925 03:05:14.127194 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c23337a7cb74184a0e57b7095cb6a11" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:14.127919 24473 raft_consensus.cc:1261] T 7993f9c3e2a44f408bbf5d1e395e2306 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260925 03:05:14.153215 24543 raft_consensus.cc:2454] T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 9.
I20260925 03:05:14.154004 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e5819c15eac4544a57e5ee6c9d4f117" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:05:14.154966 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b627ecb4a5384ef392aa695e06599c39" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:14.155910 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7aa47c48404465b94a36fd819fe3c7a" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:14.131844 27377 consensus_queue.cc:1046] T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.003s
I20260925 03:05:14.126839 27363 consensus_queue.cc:1046] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.041s
I20260925 03:05:14.130280 24588 leader_election.cc:302] T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:05:14.128204 27508 leader_election.cc:288] T 4e5332a5d329493991dff994e70ea92c P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 8 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:14.134907 24547 raft_consensus.cc:1226] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Rejecting Update request from peer 1bc1f2e62ab44af995165bd5116a9d43 for earlier term 6. Current term is 7. Ops: []
I20260925 03:05:14.140442 27513 raft_consensus.cc:3037] T 9e8af2641e124f9589c5ccfdac785f63 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Advancing to term 9
I20260925 03:05:14.158308 27401 raft_consensus.cc:2781] T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260925 03:05:14.158696 27401 raft_consensus.cc:482] T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Starting leader election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:05:14.109581 27499 leader_election.cc:288] T a7aa47c48404465b94a36fd819fe3c7a P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
W20260925 03:05:14.159487 25165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 188412us
I20260925 03:05:14.160851 27146 consensus_queue.cc:1035] T 7993f9c3e2a44f408bbf5d1e395e2306 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260925 03:05:14.162217 25196 raft_consensus.cc:1226] T e92f377a79354a6ebb4eec67303c13e3 P 1bc1f2e62ab44af995165bd5116a9d43 [term 7 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 6. Current term is 7. Ops: []
I20260925 03:05:14.164340 27267 raft_consensus.cc:3037] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:05:14.165230 27538 consensus_queue.cc:1046] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260925 03:05:14.166427 27538 raft_consensus.cc:3032] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [term 6 LEADER]: Stepping down as leader of term 6
I20260925 03:05:14.166754 27538 raft_consensus.cc:729] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [term 6 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:05:14.135815 26800 leader_election.cc:288] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 7 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:05:14.135840 24619 raft_consensus.cc:2354] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 8. Current term is 9.
I20260925 03:05:14.137373 27534 raft_consensus.cc:3037] T 29637f5a4a504ce6ba29033ccd0cd716 P 1bc1f2e62ab44af995165bd5116a9d43 [term 9 FOLLOWER]: Advancing to term 10
I20260925 03:05:14.159149 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 10 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:14.177886 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06f4a7eaf6534e4986c9fd35911038e4" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:14.138517 27254 leader_election.cc:288] T 04e068a0f80941dc9138cee063f53bab P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:05:14.139134 27485 raft_consensus.cc:2781] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260925 03:05:14.131500 27230 consensus_queue.cc:1035] T 48ca6ccd98194cf2af7b5278d15bf4ea P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.003s
I20260925 03:05:14.133028 24436 leader_election.cc:302] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: 1bc1f2e62ab44af995165bd5116a9d43, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:14.134442 24545 raft_consensus.cc:2454] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 8.
I20260925 03:05:14.186203 24619 raft_consensus.cc:1261] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260925 03:05:14.159039 27401 raft_consensus.cc:3037] T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Advancing to term 9
I20260925 03:05:14.170542 24543 raft_consensus.cc:2396] T a7aa47c48404465b94a36fd819fe3c7a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 for term 9 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260925 03:05:14.120623 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:14.121160 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "420163a6cdce412ebc21c25fe3835638" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:14.104542 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d89396576884f7c8901dba54f783947" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:14.157239 27407 raft_consensus.cc:2781] T a4abe0ea757e4886be4945c5001e529a P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260925 03:05:14.060431 27434 raft_consensus.cc:482] T a4abe0ea757e4886be4945c5001e529a P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 FOLLOWER]: Starting leader election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:05:14.057605 27040 consensus_queue.cc:1035] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.062s
I20260925 03:05:14.050108 27346 consensus_queue.cc:1035] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.070s
I20260925 03:05:14.181993 24467 raft_consensus.cc:1226] T f22ddd7d1c444dc58f45a1fb0b3e02bc P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 8. Current term is 9. Ops: []
I20260925 03:05:14.200554 24547 raft_consensus.cc:1226] T 420163a6cdce412ebc21c25fe3835638 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 6. Current term is 7. Ops: []
I20260925 03:05:14.187054 24472 raft_consensus.cc:1261] T a7aa47c48404465b94a36fd819fe3c7a P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
W20260925 03:05:14.170203 27482 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.939s user 0.019s sys 0.033s
W20260925 03:05:14.189371 24436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 56434us
I20260925 03:05:14.206003 24544 raft_consensus.cc:2379] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c in current term 9: Already voted for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in this term.
I20260925 03:05:14.191601 27498 raft_consensus.cc:2726] T 0e54ca1757ee4e42956a9cebbeca4f62 P 869b0a8575b54d08ae2817fad5dd42d5 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260925 03:05:14.157279 27377 raft_consensus.cc:3032] T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c [term 8 LEADER]: Stepping down as leader of term 8
I20260925 03:05:14.226265 27377 raft_consensus.cc:729] T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c [term 8 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:05:14.227278 27377 consensus_queue.cc:260] T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:14.190925 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f68804eb5a244d87bb395d50683c6012" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:14.229017 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be4e0aa984f4f2b81a58e382e434ad5" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:05:14.229574 27339 raft_consensus.cc:504] T 7e49248643df4131a1c42e74244e05c2 P 1bc1f2e62ab44af995165bd5116a9d43 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:14.230147 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1414dcfe53f64a78ae413531eb8ec117" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:14.232290 27339 leader_election.cc:288] T 7e49248643df4131a1c42e74244e05c2 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 9 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:14.233083 25196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "293ef69caf514ac98510a68495f54eda" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 8 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43"
I20260925 03:05:14.228637 27377 raft_consensus.cc:3037] T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Advancing to term 9
I20260925 03:05:14.236289 24471 raft_consensus.cc:2454] T 0be4e0aa984f4f2b81a58e382e434ad5 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 8.
I20260925 03:05:14.194043 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e725b82985746b39c0cc8f671c95353" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
W20260925 03:05:14.127952 24368 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39368 (request call id 197) took 15330 ms (client timeout 15000 ms). Trace:
I20260925 03:05:14.208228 24545 raft_consensus.cc:1261] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 5. (term mismatch)
I20260925 03:05:14.244050 24545 pending_rounds.cc:76] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Aborting all ops after (but not including) 4
I20260925 03:05:14.244285 24545 pending_rounds.cc:97] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Aborting uncommitted NO_OP operation due to leader change: 7.5
I20260925 03:05:14.244515 24545 raft_consensus.cc:2866] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260925 03:05:14.184787 27485 raft_consensus.cc:482] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:05:14.221179 24540 raft_consensus.cc:1226] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 8. Current term is 9. Ops: []
I20260925 03:05:14.204574 27434 raft_consensus.cc:3037] T a4abe0ea757e4886be4945c5001e529a P 1bc1f2e62ab44af995165bd5116a9d43 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:05:14.223390 27499 consensus_queue.cc:1035] T 442d4a29b1d743599f4b6b4871e26668 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260925 03:05:14.224653 24437 leader_election.cc:302] T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:14.225328 27508 consensus_queue.cc:1035] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20260925 03:05:14.176322 24548 raft_consensus.cc:2454] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 8.
I20260925 03:05:14.238543 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37a157996e944db2906110d87ecb0c54" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:14.198724 24541 raft_consensus.cc:2454] T 3d89396576884f7c8901dba54f783947 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 5.
I20260925 03:05:14.239076 24466 raft_consensus.cc:1226] T 0be4e0aa984f4f2b81a58e382e434ad5 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 7. Current term is 8. Ops: []
I20260925 03:05:14.239344 25197 raft_consensus.cc:1226] T 7e49248643df4131a1c42e74244e05c2 P 1bc1f2e62ab44af995165bd5116a9d43 [term 9 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 8. Current term is 9. Ops: []
I20260925 03:05:14.196339 26800 raft_consensus.cc:2781] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260925 03:05:14.237448 24618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "420163a6cdce412ebc21c25fe3835638" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
W20260925 03:05:14.205232 27482 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.989s user 0.020s sys 0.033s
I20260925 03:05:14.237098 24436 leader_election.cc:302] T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: 1bc1f2e62ab44af995165bd5116a9d43, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:14.199386 27407 raft_consensus.cc:482] T a4abe0ea757e4886be4945c5001e529a P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Starting leader election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:05:14.255935 24547 raft_consensus.cc:1261] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 10 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 10 index: 5. (term mismatch)
I20260925 03:05:14.247882 24367 catalog_manager.cc:5495] T e92f377a79354a6ebb4eec67303c13e3 P 1bc1f2e62ab44af995165bd5116a9d43 reported cstate change: term changed from 3 to 6, leader changed from 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) to eeab93abfdb141a895002ec9f312296c (127.22.193.3). New cstate: current_term: 6 leader_uuid: "eeab93abfdb141a895002ec9f312296c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:05:14.089232 27510 raft_consensus.cc:482] T 100c113a9fd84cc8adf2ec7b33ea0093 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
W20260925 03:05:14.190903 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 152800us
W20260925 03:05:14.266419 24585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.4:43979, user_credentials={real_user=slave}} blocked reactor thread for 55464us
I20260925 03:05:14.247430 27485 raft_consensus.cc:3037] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Advancing to term 9
W20260925 03:05:14.276417 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 51858.2us
I20260925 03:05:14.277347 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5278a60f97da4e10aecd50a8d9a9a43c" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:14.293396 24437 leader_election.cc:302] T edcf6e6a422d4d198d4fbb0f7d3d845d P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
W20260925 03:05:14.281760 24436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 44733.6us
I20260925 03:05:14.282316 24546 raft_consensus.cc:2396] T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20260925 03:05:14.268348 24547 pending_rounds.cc:76] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Aborting all ops after (but not including) 4
I20260925 03:05:14.294412 24547 pending_rounds.cc:97] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Aborting uncommitted NO_OP operation due to leader change: 9.5
I20260925 03:05:14.294696 24547 raft_consensus.cc:2866] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260925 03:05:14.285467 27510 raft_consensus.cc:3037] T 100c113a9fd84cc8adf2ec7b33ea0093 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:05:14.280042 24514 leader_election.cc:302] T 420163a6cdce412ebc21c25fe3835638 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:05:14.291587 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "100c113a9fd84cc8adf2ec7b33ea0093" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:14.301878 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b8e0d260a884ecdb20f79de46c36b19" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:05:14.303100 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d89396576884f7c8901dba54f783947" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:14.304207 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "550118bc1a0641ce9803bdbf43f2069f" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:14.291592 24539 raft_consensus.cc:3037] T 7993f9c3e2a44f408bbf5d1e395e2306 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Advancing to term 7
I20260925 03:05:14.277616 24511 leader_election.cc:302] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:05:14.305258 27268 raft_consensus.cc:2781] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260925 03:05:14.305614 27268 raft_consensus.cc:482] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:14.305953 27268 raft_consensus.cc:3037] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Advancing to term 9
I20260925 03:05:14.312400 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b030d84c670b48d5bff337ccc0c77626" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 11 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:14.314050 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:05:14.317770 27479 raft_consensus.cc:2781] T 420163a6cdce412ebc21c25fe3835638 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Leader election won for term 7
I20260925 03:05:14.318724 24539 raft_consensus.cc:1261] T 7993f9c3e2a44f408bbf5d1e395e2306 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260925 03:05:14.319286 27479 raft_consensus.cc:686] T 420163a6cdce412ebc21c25fe3835638 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 LEADER]: Becoming Leader. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:05:14.326545 27479 consensus_queue.cc:237] T 420163a6cdce412ebc21c25fe3835638 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:14.328430 27346 raft_consensus.cc:2726] T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260925 03:05:14.328895 27482 raft_consensus.cc:2726] T b1e85248d3b04b979dd080caeeb02fdb P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260925 03:05:14.329234 27451 leader_election.cc:288] T f22ddd7d1c444dc58f45a1fb0b3e02bc P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 9 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:14.329545 24436 leader_election.cc:398] T 8e5819c15eac4544a57e5ee6c9d4f117 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 8 pre-election: Vote denied by peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105) with higher term. Message: Invalid argument: T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 8. Current term is 9.
I20260925 03:05:14.329859 24436 leader_election.cc:401] T 8e5819c15eac4544a57e5ee6c9d4f117 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 8 pre-election: Cancelling election due to peer responding with higher term
I20260925 03:05:14.263181 27538 consensus_queue.cc:260] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:14.330751 27538 raft_consensus.cc:3037] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:05:14.322273 27463 consensus_queue.cc:1035] T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.004s
I20260925 03:05:14.324846 24511 leader_election.cc:302] T 442d4a29b1d743599f4b6b4871e26668 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:05:14.333778 24543 raft_consensus.cc:1226] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 8. Current term is 9. Ops: []
I20260925 03:05:14.339072 24618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21c43af5e6844fdaacc8c625fa01fd8d" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:14.265568 26800 raft_consensus.cc:482] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:14.276108 27407 raft_consensus.cc:3037] T a4abe0ea757e4886be4945c5001e529a P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:05:14.344007 24541 raft_consensus.cc:1261] T a7aa47c48404465b94a36fd819fe3c7a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 5. (term mismatch)
I20260925 03:05:14.354436 24436 leader_election.cc:302] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: 1bc1f2e62ab44af995165bd5116a9d43, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:14.355856 24542 raft_consensus.cc:1226] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 8. Current term is 9. Ops: []
I20260925 03:05:14.363761 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7aa47c48404465b94a36fd819fe3c7a" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:14.352615 27267 raft_consensus.cc:2781] T edcf6e6a422d4d198d4fbb0f7d3d845d P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260925 03:05:14.318975 27268 raft_consensus.cc:504] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:14.366698 27268 leader_election.cc:288] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 9 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:05:14.364622 27267 raft_consensus.cc:482] T edcf6e6a422d4d198d4fbb0f7d3d845d P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:14.368176 27267 raft_consensus.cc:3037] T edcf6e6a422d4d198d4fbb0f7d3d845d P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Advancing to term 6
W20260925 03:05:14.368623 24511 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 43865.5us
I20260925 03:05:14.370798 24511 leader_election.cc:302] T 367277daf8d04778b3fa4ed94d71580d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:14.371212 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7691c6b475428c82d6181d693d26f3" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:14.364373 27368 raft_consensus.cc:2781] T 442d4a29b1d743599f4b6b4871e26668 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260925 03:05:14.372282 27368 raft_consensus.cc:482] T 442d4a29b1d743599f4b6b4871e26668 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:05:14.372632 27368 raft_consensus.cc:3037] T 442d4a29b1d743599f4b6b4871e26668 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Advancing to term 9
I20260925 03:05:14.368125 27543 consensus_queue.cc:1035] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.005s
I20260925 03:05:14.374997 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5332a5d329493991dff994e70ea92c" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:05:14.375722 24622 raft_consensus.cc:2354] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 8. Current term is 9.
I20260925 03:05:14.349820 24367 catalog_manager.cc:5495] T 0e54ca1757ee4e42956a9cebbeca4f62 P 1bc1f2e62ab44af995165bd5116a9d43 reported cstate change: term changed from 8 to 10, leader changed from 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1) to 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4). New cstate: current_term: 10 leader_uuid: "1bc1f2e62ab44af995165bd5116a9d43" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260925 03:05:14.389056 24367 catalog_manager.cc:5495] T 04e068a0f80941dc9138cee063f53bab P 1bc1f2e62ab44af995165bd5116a9d43 reported cstate change: term changed from 2 to 4, leader changed from 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4) to a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2). New cstate: current_term: 4 leader_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
I20260925 03:05:14.393358 24436 leader_election.cc:398] T 4e5332a5d329493991dff994e70ea92c P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 8 election: Vote denied by peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105) with higher term. Message: Invalid argument: T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 8. Current term is 9.
I20260925 03:05:14.393896 24436 leader_election.cc:401] T 4e5332a5d329493991dff994e70ea92c P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 8 election: Cancelling election due to peer responding with higher term
I20260925 03:05:14.265898 27230 consensus_queue.cc:1035] T a7aa47c48404465b94a36fd819fe3c7a P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.072s
W20260925 03:05:14.395067 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 101770us
I20260925 03:05:14.396255 24543 raft_consensus.cc:1226] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 8. Current term is 9. Ops: []
I20260925 03:05:14.367658 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f22ddd7d1c444dc58f45a1fb0b3e02bc" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:05:14.354365 24541 pending_rounds.cc:76] T a7aa47c48404465b94a36fd819fe3c7a P a04a37fc4aa7460cbfdbeeb4e451e1f5: Aborting all ops after (but not including) 4
I20260925 03:05:14.400728 24541 pending_rounds.cc:97] T a7aa47c48404465b94a36fd819fe3c7a P a04a37fc4aa7460cbfdbeeb4e451e1f5: Aborting uncommitted NO_OP operation due to leader change: 7.5
I20260925 03:05:14.400867 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25102ba08e4e49388f6176badcd10a25" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 8 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:14.401326 24541 raft_consensus.cc:2866] T a7aa47c48404465b94a36fd819fe3c7a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260925 03:05:14.402573 27543 raft_consensus.cc:2726] T 367277daf8d04778b3fa4ed94d71580d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260925 03:05:14.403648 24511 leader_election.cc:302] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:05:14.404982 27126 raft_consensus.cc:2781] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260925 03:05:14.405360 27126 raft_consensus.cc:482] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:05:14.405670 27126 raft_consensus.cc:3037] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Advancing to term 9
W20260925 03:05:14.407088 27553 log.cc:924] Time spent T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.058s user 0.001s sys 0.000s
I20260925 03:05:14.407672 27543 raft_consensus.cc:482] T a7aa47c48404465b94a36fd819fe3c7a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:14.408179 27414 raft_consensus.cc:482] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:14.409298 27414 raft_consensus.cc:504] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:14.411332 27414 leader_election.cc:288] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:14.408179 27543 raft_consensus.cc:504] T a7aa47c48404465b94a36fd819fe3c7a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:14.412026 27533 consensus_queue.cc:1046] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: INVALID_TERM, Last received: 5.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260925 03:05:14.388855 27503 consensus_queue.cc:1046] T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.050s
I20260925 03:05:14.413507 27543 leader_election.cc:288] T a7aa47c48404465b94a36fd819fe3c7a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:14.416043 27533 raft_consensus.cc:3032] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43 [term 7 LEADER]: Stepping down as leader of term 7
I20260925 03:05:14.416471 27533 raft_consensus.cc:729] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 1bc1f2e62ab44af995165bd5116a9d43, State: Running, Role: LEADER
I20260925 03:05:14.417351 27533 consensus_queue.cc:260] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 7.9, Last appended by leader: 9, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:14.418618 27533 raft_consensus.cc:3037] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43 [term 7 FOLLOWER]: Advancing to term 8
I20260925 03:05:14.247198 27022 consensus_queue.cc:260] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:14.353417 26800 raft_consensus.cc:3037] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Advancing to term 9
I20260925 03:05:14.412504 24613 raft_consensus.cc:1261] T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c [term 10 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 10 index: 5. (index mismatch)
I20260925 03:05:14.424391 24542 raft_consensus.cc:1226] T 0be4e0aa984f4f2b81a58e382e434ad5 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 7. Current term is 8. Ops: []
I20260925 03:05:14.415586 27267 raft_consensus.cc:504] T edcf6e6a422d4d198d4fbb0f7d3d845d P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:05:14.243301 24368 rpcz_store.cc:269] 0925 03:04:58.797286 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:04:58.797433 (+ 147us) service_pool.cc:227] Handling call
0925 03:04:59.038343 (+240910us) spinlock_profiling.cc:243] Waited 22.6 ms on lock 0x7b54001509d4. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455af0 00007f43764d8f2a 00007f437c42ad6c 00007f437c524c3e 00007f437a836fb4 00007f437a836f42 00007f437a836e71 00007f437a836dec 00007f437a8360a2 00007f43748fa94c 00007f43748f9d86
0925 03:05:14.127687 (+15089344us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0925 03:05:09.136980 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0925 03:05:09.137477 (+ 497us) write_op.cc:432] Acquiring schema lock in shared mode
0925 03:05:09.137530 (+ 53us) write_op.cc:435] Acquired schema lock
0925 03:05:09.137548 (+ 18us) tablet.cc:660] Decoding operations
0925 03:05:09.138876 (+ 1328us) write_op.cc:620] Acquiring the partition lock for write op
0925 03:05:09.138935 (+ 59us) write_op.cc:641] Partition lock acquired for write op
0925 03:05:09.138971 (+ 36us) tablet.cc:683] Acquiring locks for 15 operations
0925 03:05:09.139946 (+ 975us) tablet.cc:699] Row locks acquired
0925 03:05:09.139977 (+ 31us) write_op.cc:260] PREPARE: finished
0925 03:05:09.140121 (+ 144us) write_op.cc:270] Start()
0925 03:05:09.140225 (+ 104us) write_op.cc:276] Timestamp: P: 1790305509140073 usec, L: 0
0925 03:05:09.188527 (+ 48302us) op_driver.cc:331] REPLICATION: starting
0925 03:05:09.248183 (+ 59656us) log.cc:842] Serialized 5898 byte log entry
0925 03:05:10.609929 (+1361746us) op_driver.cc:444] REPLICATION: finished
0925 03:05:10.942081 (+332152us) write_op.cc:301] APPLY: starting
0925 03:05:10.942181 (+ 100us) tablet.cc:1365] starting BulkCheckPresence
0925 03:05:10.942441 (+ 260us) tablet.cc:1368] finished BulkCheckPresence
0925 03:05:10.942563 (+ 122us) tablet.cc:1370] starting ApplyRowOperation cycle
0925 03:05:10.944204 (+ 1641us) tablet.cc:1381] finished ApplyRowOperation cycle
0925 03:05:10.944467 (+ 263us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=15
0925 03:05:10.944521 (+ 54us) write_op.cc:312] APPLY: finished
0925 03:05:10.999083 (+ 54562us) log.cc:842] Serialized 106 byte log entry
0925 03:05:11.047272 (+ 48189us) write_op.cc:489] Releasing partition, row and schema locks
0925 03:05:11.069697 (+ 22425us) write_op.cc:454] Released schema lock
0925 03:05:11.079151 (+ 9454us) write_op.cc:341] FINISH: Updating metrics
0925 03:05:11.116024 (+ 36873us) write_op.cc:489] Releasing partition, row and schema locks
0925 03:05:11.120864 (+ 4840us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":76594,"reported_tablets":48,"spinlock_wait_cycles":51870592,"child_traces":[["op",{"apply.queue_time_us":276069,"apply.run_cpu_time_us":5018,"apply.run_wall_time_us":211558,"num_ops":15,"prepare.queue_time_us":157846,"prepare.run_cpu_time_us":69872,"prepare.run_wall_time_us":265770,"raft.queue_time_us":657620,"raft.run_cpu_time_us":34461,"raft.run_wall_time_us":163513,"replication_time_us":1421285,"thread_start_us":509382,"threads_started":4,"wal-append.queue_time_us":256365,"wal-append.run_cpu_time_us":23855,"wal-append.run_wall_time_us":2630802}]]}
I20260925 03:05:14.439750 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e068a0f80941dc9138cee063f53bab" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
W20260925 03:05:14.440924 24585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.4:43979, user_credentials={real_user=slave}} blocked reactor thread for 48836us
I20260925 03:05:14.441390 27328 consensus_queue.cc:1046] T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260925 03:05:14.390338 27427 leader_election.cc:288] T 25102ba08e4e49388f6176badcd10a25 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:14.439028 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e92f377a79354a6ebb4eec67303c13e3" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:05:14.443953 27267 leader_election.cc:288] T edcf6e6a422d4d198d4fbb0f7d3d845d P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:14.442535 27328 raft_consensus.cc:3032] T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c [term 8 LEADER]: Stepping down as leader of term 8
I20260925 03:05:14.349905 27482 raft_consensus.cc:2726] T 8e5819c15eac4544a57e5ee6c9d4f117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Leader pre-election lost for term 8. Reason: Vote denied by peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105) with higher term. Message: Invalid argument: T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 8. Current term is 9.
W20260925 03:05:14.415055 27560 log.cc:924] Time spent T 11bb46bce3564841b7674e6a693bd2e6 P 869b0a8575b54d08ae2817fad5dd42d5: Append to log took a long time: real 0.052s user 0.000s sys 0.000s
I20260925 03:05:14.399235 27363 raft_consensus.cc:3037] T 4e5332a5d329493991dff994e70ea92c P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Advancing to term 9
I20260925 03:05:14.448804 24367 catalog_manager.cc:5495] T 8ac466793d094325ace224708b2158ed P 1bc1f2e62ab44af995165bd5116a9d43 reported cstate change: term changed from 5 to 6, leader changed from eeab93abfdb141a895002ec9f312296c (127.22.193.3) to 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4). New cstate: current_term: 6 leader_uuid: "1bc1f2e62ab44af995165bd5116a9d43" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260925 03:05:14.395042 27346 raft_consensus.cc:2726] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260925 03:05:14.453615 24511 leader_election.cc:302] T 0b5df303cd6a4d59aca5c18bbbff24ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:14.399883 27403 consensus_queue.cc:1046] T f22ddd7d1c444dc58f45a1fb0b3e02bc P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.055s
I20260925 03:05:14.456509 27022 raft_consensus.cc:3037] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Advancing to term 9
I20260925 03:05:14.457156 24618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcf6e6a422d4d198d4fbb0f7d3d845d" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:05:14.457935 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:05:14.454994 27499 consensus_queue.cc:1035] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.002s
W20260925 03:05:14.455284 27481 consensus_meta.cc:287] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent flushing consensus metadata: real 0.508s user 0.010s sys 0.000s
I20260925 03:05:14.449599 27328 raft_consensus.cc:729] T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c [term 8 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:05:14.478695 24367 catalog_manager.cc:5495] T a4abe0ea757e4886be4945c5001e529a P 1bc1f2e62ab44af995165bd5116a9d43 reported cstate change: term changed from 2 to 3, leader changed from eeab93abfdb141a895002ec9f312296c (127.22.193.3) to a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2). New cstate: current_term: 3 leader_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } } }
W20260925 03:05:14.481766 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 81658.6us
I20260925 03:05:14.486093 24511 leader_election.cc:302] T ac9357d144aa49789faaf31fe5e6bf89 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:14.487366 24511 leader_election.cc:302] T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 1bc1f2e62ab44af995165bd5116a9d43, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:14.487739 27126 raft_consensus.cc:504] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:14.488798 27414 raft_consensus.cc:2726] T ac9357d144aa49789faaf31fe5e6bf89 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260925 03:05:14.489430 27414 raft_consensus.cc:3037] T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:05:14.488502 24511 leader_election.cc:302] T c85f5d1d3f004cf5817bb594fd8e6469 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:14.454458 27385 leader_election.cc:288] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 9 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:05:14.461825 27463 raft_consensus.cc:2726] T 0b5df303cd6a4d59aca5c18bbbff24ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260925 03:05:14.447100 27401 raft_consensus.cc:504] T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:14.503116 24437 leader_election.cc:302] T c85f5d1d3f004cf5817bb594fd8e6469 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:05:14.453791 27369 raft_consensus.cc:504] T 1414dcfe53f64a78ae413531eb8ec117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:14.522310 24511 leader_election.cc:302] T edcf6e6a422d4d198d4fbb0f7d3d845d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:14.523859 24511 leader_election.cc:302] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:14.524881 27268 raft_consensus.cc:2726] T edcf6e6a422d4d198d4fbb0f7d3d845d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260925 03:05:14.524978 27505 consensus_queue.cc:1046] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260925 03:05:14.525616 27268 raft_consensus.cc:2726] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260925 03:05:14.526022 27456 raft_consensus.cc:3032] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 LEADER]: Stepping down as leader of term 7
I20260925 03:05:14.526408 27456 raft_consensus.cc:729] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:05:14.531816 24470 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1481) took 1389 ms. Trace:
I20260925 03:05:14.516181 27403 raft_consensus.cc:2726] T c85f5d1d3f004cf5817bb594fd8e6469 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260925 03:05:14.532263 24470 rpcz_store.cc:276] 0925 03:05:13.142482 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:05:13.847194 (+704712us) service_pool.cc:227] Handling call
0925 03:05:13.930680 (+ 83486us) spinlock_profiling.cc:243] Waited 3.07 ms on lock 0x7b5800001598. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f43733ecf11 00007f4373570851 00007f4373570702 00007f437c460b09 00007f437b39c730 00007f437b46279b 00007f437b462821 00007f437b4262b0 00007f437b4117f9 00007f437b4112a2 00007f4374af7684
0925 03:05:13.976621 (+ 45941us) spinlock_profiling.cc:243] Waited 41 ms on lock 0x7b5800001598. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f43733ecf11 00007f4373570b7c 00007f437357073b 00007f437c460b09 00007f437b39c730 00007f437b46279b 00007f437b462821 00007f437b4262b0 00007f437b4117f9 00007f437b4112a2 00007f4374af7684
0925 03:05:13.976810 (+ 189us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:05:13.977168 (+ 358us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:05:13.977194 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:05:13.977854 (+ 660us) log.cc:842] Serialized 26 byte log entry
0925 03:05:14.250353 (+272499us) raft_consensus.cc:1631] Marking committed up to 2
0925 03:05:14.250410 (+ 57us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:05:14.250497 (+ 87us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:05:14.414457 (+163960us) spinlock_profiling.cc:243] Waited 29 ms on lock 0x7b44000232cc. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f43733ecf11 00007f43734baa95 00007f43734bb4b2 00007f4376163511 00007f437615ee1a 00007f4376152bee 00007f4376151f36 00007f4376151b58 00007f437615b551 00007f437615b509 00007f437615b499
0925 03:05:14.531588 (+117131us) raft_consensus.cc:1671] finished
0925 03:05:14.531609 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:05:14.531757 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":4296,"raft.run_cpu_time_us":185,"raft.run_wall_time_us":13342,"spinlock_wait_cycles":167990784,"thread_start_us":272004,"threads_started":1,"wal-append.queue_time_us":384947}
I20260925 03:05:14.534682 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5332a5d329493991dff994e70ea92c" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:14.534938 27485 raft_consensus.cc:504] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:14.533418 27482 raft_consensus.cc:2781] T c85f5d1d3f004cf5817bb594fd8e6469 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260925 03:05:14.537096 27482 raft_consensus.cc:482] T c85f5d1d3f004cf5817bb594fd8e6469 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:14.537436 27482 raft_consensus.cc:3037] T c85f5d1d3f004cf5817bb594fd8e6469 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:05:14.483301 27481 raft_consensus.cc:504] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:14.488377 27429 consensus_queue.cc:1046] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.058s
W20260925 03:05:14.531931 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 47396.1us
I20260925 03:05:14.573582 24546 raft_consensus.cc:1226] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 7. Current term is 8. Ops: []
I20260925 03:05:14.533099 24367 catalog_manager.cc:5495] T b4ebd2b043024a548005c8b6b00b0449 P 1bc1f2e62ab44af995165bd5116a9d43 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "1bc1f2e62ab44af995165bd5116a9d43" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260925 03:05:14.572918 24437 leader_election.cc:302] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
W20260925 03:05:14.576149 24511 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 46961.7us
I20260925 03:05:14.577533 24511 leader_election.cc:302] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:14.578820 24511 leader_election.cc:302] T 43366a0a2f9d4344bf4e37ffd290ef8a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:14.581502 24540 raft_consensus.cc:1226] T 1414dcfe53f64a78ae413531eb8ec117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 4. Current term is 5. Ops: []
I20260925 03:05:14.577090 24620 raft_consensus.cc:2454] T 420163a6cdce412ebc21c25fe3835638 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 7.
I20260925 03:05:14.585572 24367 catalog_manager.cc:5495] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43 reported cstate change: term changed from 4 to 6, leader changed from 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1) to 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4). New cstate: current_term: 6 leader_uuid: "1bc1f2e62ab44af995165bd5116a9d43" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260925 03:05:14.589927 24511 leader_election.cc:302] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:14.593761 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ac285175464fbda67211d9574db415" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:05:14.594395 24613 raft_consensus.cc:3037] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Advancing to term 9
I20260925 03:05:14.595101 27369 leader_election.cc:288] T 1414dcfe53f64a78ae413531eb8ec117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:14.571668 27456 consensus_queue.cc:260] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:14.596495 27456 raft_consensus.cc:3037] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Advancing to term 8
I20260925 03:05:14.597612 27505 consensus_queue.cc:1035] T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20260925 03:05:14.591347 27508 raft_consensus.cc:2781] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260925 03:05:14.601106 27508 raft_consensus.cc:482] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:14.601471 27508 raft_consensus.cc:3037] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Advancing to term 8
I20260925 03:05:14.601938 27414 raft_consensus.cc:2726] T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260925 03:05:14.591573 27403 raft_consensus.cc:3037] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Advancing to term 9
I20260925 03:05:14.606084 27481 leader_election.cc:288] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 8 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:14.607599 24367 catalog_manager.cc:5495] T 71f461a6e2ff434993b78a2d077f0abd P 1bc1f2e62ab44af995165bd5116a9d43 reported cstate change: term changed from 3 to 8. New cstate: current_term: 8 leader_uuid: "1bc1f2e62ab44af995165bd5116a9d43" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260925 03:05:14.612298 24511 leader_election.cc:302] T 37a157996e944db2906110d87ecb0c54 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:14.616062 24437 leader_election.cc:302] T b4ebd2b043024a548005c8b6b00b0449 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: 1bc1f2e62ab44af995165bd5116a9d43, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:05:14.617416 24437 leader_election.cc:302] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: 1bc1f2e62ab44af995165bd5116a9d43, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:05:14.587209 27268 raft_consensus.cc:2726] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260925 03:05:14.541180 27513 raft_consensus.cc:504] T 9e8af2641e124f9589c5ccfdac785f63 P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:14.618616 24436 leader_election.cc:302] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:05:14.630151 27407 raft_consensus.cc:504] T a4abe0ea757e4886be4945c5001e529a P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:14.638314 27508 raft_consensus.cc:504] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:14.640550 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "293ef69caf514ac98510a68495f54eda" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 8 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:14.653565 24540 raft_consensus.cc:2379] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 in current term 8: Already voted for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in this term.
I20260925 03:05:14.642241 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5332a5d329493991dff994e70ea92c" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:05:14.645637 25163 leader_election.cc:302] T 04e068a0f80941dc9138cee063f53bab P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:14.646628 27403 raft_consensus.cc:2726] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260925 03:05:14.607525 27401 leader_election.cc:288] T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 9 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:05:14.587044 27485 leader_election.cc:288] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 9 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:14.650714 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abd5b8dc1b7e49e3aa9af900d73ff0e7" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
W20260925 03:05:14.657013 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 534399us
I20260925 03:05:14.660887 25163 leader_election.cc:302] T e92f377a79354a6ebb4eec67303c13e3 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:14.614735 27414 raft_consensus.cc:2726] T 37a157996e944db2906110d87ecb0c54 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260925 03:05:14.673347 24619 raft_consensus.cc:1226] T a4abe0ea757e4886be4945c5001e529a P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 3. Current term is 4. Ops: []
I20260925 03:05:14.674255 25163 leader_election.cc:302] T 7e49248643df4131a1c42e74244e05c2 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:14.664435 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b627ecb4a5384ef392aa695e06599c39" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:14.664881 24437 leader_election.cc:302] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:05:14.618868 27346 raft_consensus.cc:2726] T b4ebd2b043024a548005c8b6b00b0449 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260925 03:05:14.666246 24436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 55875.1us
I20260925 03:05:14.659861 27427 raft_consensus.cc:2741] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20260925 03:05:14.672207 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06f4a7eaf6534e4986c9fd35911038e4" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:14.648731 27267 raft_consensus.cc:2726] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260925 03:05:14.675973 24544 raft_consensus.cc:2454] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 8.
I20260925 03:05:14.703274 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:05:14.715728 24543 raft_consensus.cc:2354] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 8. Current term is 9.
I20260925 03:05:14.721390 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1414dcfe53f64a78ae413531eb8ec117" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:05:14.609998 27534 raft_consensus.cc:504] T 29637f5a4a504ce6ba29033ccd0cd716 P 1bc1f2e62ab44af995165bd5116a9d43 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:14.667130 27533 raft_consensus.cc:2726] T 04e068a0f80941dc9138cee063f53bab P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260925 03:05:14.736677 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7aa47c48404465b94a36fd819fe3c7a" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:14.699148 27339 raft_consensus.cc:2726] T e92f377a79354a6ebb4eec67303c13e3 P 1bc1f2e62ab44af995165bd5116a9d43 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260925 03:05:14.698975 27254 raft_consensus.cc:2726] T 7e49248643df4131a1c42e74244e05c2 P 1bc1f2e62ab44af995165bd5116a9d43 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
W20260925 03:05:14.753549 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 88756.9us
I20260925 03:05:14.755409 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f68804eb5a244d87bb395d50683c6012" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:05:14.754513 24437 leader_election.cc:302] T 100c113a9fd84cc8adf2ec7b33ea0093 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:05:14.722234 27414 raft_consensus.cc:482] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:14.614909 27328 consensus_queue.cc:260] T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:14.775938 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5332a5d329493991dff994e70ea92c" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:05:14.765501 27414 raft_consensus.cc:504] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:14.777176 27368 raft_consensus.cc:504] T 442d4a29b1d743599f4b6b4871e26668 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:14.778251 27414 leader_election.cc:288] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:14.778908 27126 leader_election.cc:288] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 9 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:14.739194 27346 raft_consensus.cc:2781] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260925 03:05:14.793350 24437 leader_election.cc:302] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:14.794839 24469 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55560 (request call id 1546) took 1565 ms. Trace:
I20260925 03:05:14.797338 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06f4a7eaf6534e4986c9fd35911038e4" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:05:14.798146 24546 raft_consensus.cc:2379] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c in current term 9: Already voted for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in this term.
I20260925 03:05:14.795212 24469 rpcz_store.cc:276] 0925 03:05:13.229742 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:05:13.973017 (+743275us) service_pool.cc:227] Handling call
0925 03:05:13.973299 (+ 282us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:05:13.973491 (+ 192us) raft_consensus.cc:1520] Early marking committed up to index 6
0925 03:05:14.261302 (+287811us) log.cc:842] Serialized 14 byte log entry
0925 03:05:14.794511 (+533209us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:05:14.794558 (+ 47us) raft_consensus.cc:1631] Marking committed up to 6
0925 03:05:14.794605 (+ 47us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:05:14.794678 (+ 73us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:05:14.794789 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":31804,"prepare.queue_time_us":483002,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":44,"thread_start_us":696006,"threads_started":2}
I20260925 03:05:14.760921 26800 raft_consensus.cc:504] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:14.803918 27534 leader_election.cc:288] T 29637f5a4a504ce6ba29033ccd0cd716 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 10 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:14.754465 27403 raft_consensus.cc:482] T cc2df40db08649c9a75e90000b878559 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:14.782397 27363 raft_consensus.cc:2726] T 4e5332a5d329493991dff994e70ea92c P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Leader election lost for term 8. Reason: Vote denied by peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105) with higher term. Message: Invalid argument: T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 8. Current term is 9.
I20260925 03:05:14.779207 27368 leader_election.cc:288] T 442d4a29b1d743599f4b6b4871e26668 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 9 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:14.812457 24545 raft_consensus.cc:1226] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 8. Current term is 9. Ops: []
I20260925 03:05:14.804266 26800 leader_election.cc:288] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 9 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:14.817579 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
W20260925 03:05:14.805727 27434 consensus_meta.cc:287] T a4abe0ea757e4886be4945c5001e529a P 1bc1f2e62ab44af995165bd5116a9d43: Time spent flushing consensus metadata: real 0.540s user 0.005s sys 0.004s
I20260925 03:05:14.793393 27346 raft_consensus.cc:482] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
W20260925 03:05:14.818291 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 46226.1us
I20260925 03:05:14.830394 24437 leader_election.cc:302] T cc2df40db08649c9a75e90000b878559 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:14.832371 24437 leader_election.cc:302] T e6b6ec010caf4689b8c90449764f5cf9 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: 1bc1f2e62ab44af995165bd5116a9d43, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:05:14.770143 27498 raft_consensus.cc:2781] T 100c113a9fd84cc8adf2ec7b33ea0093 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260925 03:05:14.758730 27510 raft_consensus.cc:504] T 100c113a9fd84cc8adf2ec7b33ea0093 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:14.878872 24613 raft_consensus.cc:2454] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 9.
I20260925 03:05:14.880426 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06f4a7eaf6534e4986c9fd35911038e4" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:14.883788 24436 leader_election.cc:302] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:05:14.819059 27555 raft_consensus.cc:2726] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260925 03:05:14.889114 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 105477us
I20260925 03:05:14.835871 27346 raft_consensus.cc:3037] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Advancing to term 9
I20260925 03:05:14.892616 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442d4a29b1d743599f4b6b4871e26668" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:05:14.894366 24586 leader_election.cc:302] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 1bc1f2e62ab44af995165bd5116a9d43, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:05:14.869997 27498 raft_consensus.cc:482] T 100c113a9fd84cc8adf2ec7b33ea0093 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:14.895807 27230 raft_consensus.cc:3032] T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c [term 7 LEADER]: Stepping down as leader of term 7
I20260925 03:05:14.896248 27230 raft_consensus.cc:729] T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c [term 7 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:05:14.896313 27543 raft_consensus.cc:3032] T f22ddd7d1c444dc58f45a1fb0b3e02bc P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 LEADER]: Stepping down as leader of term 8
I20260925 03:05:14.896790 27543 raft_consensus.cc:729] T f22ddd7d1c444dc58f45a1fb0b3e02bc P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
W20260925 03:05:14.897017 27562 log.cc:924] Time spent T 25102ba08e4e49388f6176badcd10a25 P 869b0a8575b54d08ae2817fad5dd42d5: Append to log took a long time: real 0.098s user 0.001s sys 0.000s
I20260925 03:05:14.897294 27230 consensus_queue.cc:260] T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:14.897693 27543 consensus_queue.cc:260] T f22ddd7d1c444dc58f45a1fb0b3e02bc P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:14.898409 27230 raft_consensus.cc:3037] T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Advancing to term 8
I20260925 03:05:14.898900 27384 consensus_queue.cc:1035] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260925 03:05:14.903505 24468 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1482) took 1760 ms. Trace:
I20260925 03:05:14.903919 24468 rpcz_store.cc:276] 0925 03:05:13.142877 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:05:13.854396 (+711519us) service_pool.cc:227] Handling call
0925 03:05:13.997554 (+143158us) raft_consensus.cc:1457] Updating replica for 3 ops
0925 03:05:14.064283 (+ 66729us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:05:14.064314 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 3 ops
0925 03:05:14.104773 (+ 40459us) log.cc:842] Serialized 78 byte log entry
0925 03:05:14.467621 (+362848us) raft_consensus.cc:1631] Marking committed up to 3
0925 03:05:14.505144 (+ 37523us) log.cc:842] Serialized 14 byte log entry
0925 03:05:14.512314 (+ 7170us) log.cc:842] Serialized 14 byte log entry
0925 03:05:14.512567 (+ 253us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:05:14.625311 (+112744us) spinlock_profiling.cc:243] Waited 58.5 ms on lock 0x7b54003fe654. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d0f51 00007f43764cf99e 00007f437b4112e9 00007f4374af7684 00007f4374af7612 00007f4374af7541 00007f4374af74bc 00007f4374af6772 00007f43748fa94c 00007f43748f9d86
0925 03:05:14.625359 (+ 48us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:05:14.902132 (+276773us) spinlock_profiling.cc:243] Waited 5.78 ms on lock 0x7b48000c0cd8. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f437b3777f1 00007f437b36e8e7 00007f437b36e77a 00007f437b46295f 00007f437b462627 00007f437b476b2c 00007f437b476ad8 00007f437b476a58 00007f437b476a13 00007f437b475cbc 00007f437a3f7be6
0925 03:05:14.903240 (+ 1108us) raft_consensus.cc:1671] finished
0925 03:05:14.903260 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:05:14.903443 (+ 183us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":26037,"prepare.queue_time_us":101192,"prepare.run_cpu_time_us":97,"prepare.run_wall_time_us":2670,"raft.queue_time_us":28143,"raft.run_cpu_time_us":3382,"raft.run_wall_time_us":424171,"spinlock_wait_cycles":147763840,"thread_start_us":310521,"threads_started":1,"wal-append.queue_time_us":580269}
I20260925 03:05:14.905803 24465 raft_consensus.cc:1203] T 25102ba08e4e49388f6176badcd10a25 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[3.2-6.4] Dedup: 5.3->[6.4-6.4]
I20260925 03:05:14.906198 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:14.906876 24468 raft_consensus.cc:3037] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Advancing to term 10
I20260925 03:05:14.906366 24465 raft_consensus.cc:1226] T 25102ba08e4e49388f6176badcd10a25 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 6. Current term is 7. Ops: [3.2-6.4]
I20260925 03:05:14.904495 27482 raft_consensus.cc:504] T c85f5d1d3f004cf5817bb594fd8e6469 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:14.910247 27494 raft_consensus.cc:3032] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 LEADER]: Stepping down as leader of term 6
I20260925 03:05:14.910596 27494 raft_consensus.cc:729] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 1bc1f2e62ab44af995165bd5116a9d43, State: Running, Role: LEADER
I20260925 03:05:14.911661 27407 leader_election.cc:288] T a4abe0ea757e4886be4945c5001e529a P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:05:14.911484 27494 consensus_queue.cc:260] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:14.915663 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:05:14.916153 27368 consensus_queue.cc:1046] T a4abe0ea757e4886be4945c5001e529a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260925 03:05:14.916723 27485 raft_consensus.cc:2781] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Leader election won for term 9
I20260925 03:05:14.917300 27485 raft_consensus.cc:686] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 LEADER]: Becoming Leader. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:05:14.917488 27368 raft_consensus.cc:3032] T a4abe0ea757e4886be4945c5001e529a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 LEADER]: Stepping down as leader of term 3
I20260925 03:05:14.917930 27368 raft_consensus.cc:729] T a4abe0ea757e4886be4945c5001e529a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:05:14.895840 27498 raft_consensus.cc:3037] T 100c113a9fd84cc8adf2ec7b33ea0093 P 869b0a8575b54d08ae2817fad5dd42d5 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:05:14.918740 27368 consensus_queue.cc:260] T a4abe0ea757e4886be4945c5001e529a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:14.920219 27368 raft_consensus.cc:3037] T a4abe0ea757e4886be4945c5001e529a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:05:14.918839 24465 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55560 (request call id 1547) took 1688 ms. Trace:
I20260925 03:05:14.916831 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7691c6b475428c82d6181d693d26f3" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:05:14.869107 27267 raft_consensus.cc:2726] T cc2df40db08649c9a75e90000b878559 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260925 03:05:14.921914 24465 rpcz_store.cc:276] 0925 03:05:13.230051 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:05:13.976877 (+746826us) service_pool.cc:227] Handling call
0925 03:05:14.024850 (+ 47973us) spinlock_profiling.cc:243] Waited 6.21 ms on lock 0x7b540037e7a8. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f437b3777f1 00007f437b3f22dc 00007f437b42637c 00007f437b41180b 00007f437b4112a2 00007f4374af7684 00007f4374af7612 00007f4374af7541 00007f4374af74bc 00007f4374af6772 00007f43748fa94c
0925 03:05:14.905619 (+880769us) raft_consensus.cc:1457] Updating replica for 3 ops
0925 03:05:14.909012 (+ 3393us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:05:14.918634 (+ 9622us) spinlock_profiling.cc:243] Waited 881 ms on lock 0x7b54003fe650. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764cfa09 00007f437b4112e9 00007f4374af7684 00007f4374af7612 00007f4374af7541 00007f4374af74bc 00007f4374af6772 00007f43748fa94c 00007f43748f9d86 00007f43748f972f
0925 03:05:14.918775 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2039569920}
I20260925 03:05:14.819433 27053 consensus_queue.cc:1035] T 25102ba08e4e49388f6176badcd10a25 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.070s
I20260925 03:05:14.930210 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f1fac028a144058b92331c0d9b44ebf" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:05:14.931867 24586 leader_election.cc:302] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
W20260925 03:05:14.893646 24621 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.501s user 0.012s sys 0.021s
W20260925 03:05:14.935086 24621 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.563s user 0.012s sys 0.022s
I20260925 03:05:14.820120 27403 raft_consensus.cc:504] T cc2df40db08649c9a75e90000b878559 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:14.962854 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9c33fd100ff44c1a3247319a324a044" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
W20260925 03:05:14.962857 24548 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.573s user 0.020s sys 0.039s
I20260925 03:05:14.964103 24613 raft_consensus.cc:2454] T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 9.
W20260925 03:05:14.945497 24547 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.579s user 0.015s sys 0.018s
W20260925 03:05:14.964758 24547 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.598s user 0.015s sys 0.018s
I20260925 03:05:14.965356 24511 leader_election.cc:302] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
W20260925 03:05:14.967133 24617 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.523s user 0.035s sys 0.015s
W20260925 03:05:14.967454 24617 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.523s user 0.035s sys 0.015s
W20260925 03:05:14.963603 24548 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.574s user 0.020s sys 0.039s
W20260925 03:05:14.968945 24464 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.509s user 0.035s sys 0.075s
I20260925 03:05:14.835633 27434 raft_consensus.cc:504] T a4abe0ea757e4886be4945c5001e529a P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:14.911486 27482 leader_election.cc:288] T c85f5d1d3f004cf5817bb594fd8e6469 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
W20260925 03:05:14.971221 24464 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.511s user 0.035s sys 0.075s
I20260925 03:05:14.893193 27513 leader_election.cc:288] T 9e8af2641e124f9589c5ccfdac785f63 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 9 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:14.886626 27427 consensus_queue.cc:1035] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.081s
I20260925 03:05:14.979297 24619 raft_consensus.cc:1226] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 8. Current term is 9. Ops: []
W20260925 03:05:14.980134 24471 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.529s user 0.019s sys 0.022s
W20260925 03:05:14.980510 24471 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.530s user 0.020s sys 0.022s
I20260925 03:05:14.975461 24464 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1497) took 1533 ms. Trace:
I20260925 03:05:14.975512 25197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4abe0ea757e4886be4945c5001e529a" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43"
I20260925 03:05:14.974524 24586 leader_election.cc:302] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:05:14.943343 27494 raft_consensus.cc:3037] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:05:14.926043 27508 leader_election.cc:288] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 8 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:14.983333 24586 leader_election.cc:302] T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:05:14.945128 27503 raft_consensus.cc:2726] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260925 03:05:14.921015 27363 consensus_queue.cc:1035] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.082s
I20260925 03:05:14.969363 27463 raft_consensus.cc:2726] T 43366a0a2f9d4344bf4e37ffd290ef8a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260925 03:05:14.968977 27414 raft_consensus.cc:2781] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260925 03:05:14.983757 24464 rpcz_store.cc:276] 0925 03:05:13.441944 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:05:14.017186 (+575242us) service_pool.cc:227] Handling call
0925 03:05:14.017465 (+ 279us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:05:14.017652 (+ 187us) raft_consensus.cc:1520] Early marking committed up to index 4
0925 03:05:14.459626 (+441974us) log.cc:842] Serialized 14 byte log entry
0925 03:05:14.975143 (+515517us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:05:14.975197 (+ 54us) raft_consensus.cc:1631] Marking committed up to 4
0925 03:05:14.975240 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:05:14.975297 (+ 57us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:05:14.975407 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":439108,"prepare.run_cpu_time_us":29,"prepare.run_wall_time_us":30,"raft.queue_time_us":87,"thread_start_us":919514,"threads_started":2}
I20260925 03:05:15.000823 24464 raft_consensus.cc:1226] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 7. Current term is 8. Ops: []
I20260925 03:05:14.997421 24586 leader_election.cc:302] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:05:14.979852 27328 raft_consensus.cc:3037] T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Advancing to term 9
I20260925 03:05:15.009689 24586 leader_election.cc:302] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:05:14.987205 27022 raft_consensus.cc:2726] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260925 03:05:14.997686 25191 raft_consensus.cc:1226] T a4abe0ea757e4886be4945c5001e529a P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 3. Current term is 4. Ops: []
I20260925 03:05:14.995716 27543 raft_consensus.cc:3037] T f22ddd7d1c444dc58f45a1fb0b3e02bc P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Advancing to term 9
I20260925 03:05:15.027024 27515 raft_consensus.cc:2726] T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260925 03:05:14.998647 27414 raft_consensus.cc:482] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:05:15.028290 24586 leader_election.cc:302] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters: 1bc1f2e62ab44af995165bd5116a9d43, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:05:14.992549 26800 consensus_queue.cc:1046] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.034s
W20260925 03:05:14.989692 27451 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.547s user 0.024s sys 0.030s
I20260925 03:05:15.015957 27385 raft_consensus.cc:2781] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260925 03:05:15.033572 24469 raft_consensus.cc:2454] T 06f4a7eaf6534e4986c9fd35911038e4 P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 9.
I20260925 03:05:15.037041 24588 leader_election.cc:302] T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
W20260925 03:05:15.037762 27451 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.595s user 0.024s sys 0.030s
I20260925 03:05:15.030649 27414 raft_consensus.cc:3037] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Advancing to term 10
I20260925 03:05:15.047681 24473 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38920 (request call id 229) took 1220 ms. Trace:
I20260925 03:05:15.037698 27503 raft_consensus.cc:2726] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260925 03:05:15.063056 24472 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38920 (request call id 230) took 1104 ms. Trace:
I20260925 03:05:15.043576 27515 raft_consensus.cc:2781] T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Leader election won for term 9
I20260925 03:05:15.039376 27385 raft_consensus.cc:482] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [term 10 FOLLOWER]: Starting leader election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:05:15.065819 24472 rpcz_store.cc:276] 0925 03:05:13.958642 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:05:14.206749 (+248107us) service_pool.cc:227] Handling call
0925 03:05:14.240700 (+ 33951us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:05:14.256480 (+ 15780us) raft_consensus.cc:1520] Early marking committed up to index 9
0925 03:05:14.541271 (+284791us) log.cc:842] Serialized 14 byte log entry
0925 03:05:15.062713 (+521442us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:05:15.062770 (+ 57us) raft_consensus.cc:1631] Marking committed up to 9
0925 03:05:15.062809 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:05:15.062869 (+ 60us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:05:15.062981 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":200681,"prepare.run_cpu_time_us":27,"prepare.run_wall_time_us":27,"raft.queue_time_us":7503,"thread_start_us":746127,"threads_started":2}
I20260925 03:05:15.045593 27268 raft_consensus.cc:3032] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 LEADER]: Stepping down as leader of term 8
I20260925 03:05:15.054328 27499 raft_consensus.cc:2726] T e6b6ec010caf4689b8c90449764f5cf9 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260925 03:05:15.012387 27485 consensus_queue.cc:237] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:15.074025 24468 raft_consensus.cc:2454] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 10.
W20260925 03:05:15.066864 27565 log.cc:924] Time spent T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.089s user 0.000s sys 0.001s
W20260925 03:05:15.085496 25163 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 410289us
I20260925 03:05:15.059669 24473 rpcz_store.cc:276] 0925 03:05:13.827296 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:05:14.195383 (+368087us) service_pool.cc:227] Handling call
0925 03:05:14.207644 (+ 12261us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:05:14.221747 (+ 14103us) raft_consensus.cc:1520] Early marking committed up to index 9
0925 03:05:14.578428 (+356681us) log.cc:842] Serialized 14 byte log entry
0925 03:05:15.047315 (+468887us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:05:15.047374 (+ 59us) raft_consensus.cc:1631] Marking committed up to 9
0925 03:05:15.047421 (+ 47us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:05:15.047491 (+ 70us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:05:15.047621 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":235228,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":43,"raft.queue_time_us":1779,"raft.run_cpu_time_us":92,"raft.run_wall_time_us":93,"thread_start_us":803462,"threads_started":2}
I20260925 03:05:15.088438 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "100c113a9fd84cc8adf2ec7b33ea0093" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:05:15.066068 27510 leader_election.cc:288] T 100c113a9fd84cc8adf2ec7b33ea0093 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:15.094599 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4abe0ea757e4886be4945c5001e529a" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:05:15.096071 24619 raft_consensus.cc:2379] T a4abe0ea757e4886be4945c5001e529a P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1bc1f2e62ab44af995165bd5116a9d43 in current term 4: Already voted for candidate eeab93abfdb141a895002ec9f312296c in this term.
I20260925 03:05:15.097318 27488 raft_consensus.cc:2781] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260925 03:05:15.070894 27385 raft_consensus.cc:3037] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [term 10 FOLLOWER]: Advancing to term 11
I20260925 03:05:15.052838 27252 consensus_queue.cc:1046] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.063s
I20260925 03:05:15.081777 27268 raft_consensus.cc:729] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:05:15.089833 27515 raft_consensus.cc:686] T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c [term 9 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:05:15.106940 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc2df40db08649c9a75e90000b878559" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:15.109946 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4abe0ea757e4886be4945c5001e529a" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:05:15.097746 27488 raft_consensus.cc:482] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Starting leader election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:05:15.113159 27488 raft_consensus.cc:3037] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Advancing to term 9
I20260925 03:05:15.115550 27515 consensus_queue.cc:237] T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:15.115986 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:05:15.072119 27022 consensus_queue.cc:1046] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.028s
I20260925 03:05:15.096375 27434 leader_election.cc:288] T a4abe0ea757e4886be4945c5001e529a P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 4 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:15.122820 27022 raft_consensus.cc:3032] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [term 8 LEADER]: Stepping down as leader of term 8
I20260925 03:05:15.084461 27538 consensus_queue.cc:1035] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.023s
I20260925 03:05:15.135396 27254 raft_consensus.cc:482] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:15.143807 24548 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 1520) took 1013 ms. Trace:
I20260925 03:05:15.144255 24548 rpcz_store.cc:276] 0925 03:05:14.130079 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:05:14.268242 (+138163us) service_pool.cc:227] Handling call
0925 03:05:14.268548 (+ 306us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:05:14.290207 (+ 21659us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:05:14.290238 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:05:14.290741 (+ 503us) log.cc:842] Serialized 26 byte log entry
0925 03:05:14.968084 (+677343us) raft_consensus.cc:1631] Marking committed up to 3
0925 03:05:14.968124 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:05:14.968198 (+ 74us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:05:15.143588 (+175390us) raft_consensus.cc:1671] finished
0925 03:05:15.143607 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:05:15.143728 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":15138,"raft.queue_time_us":7436,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":42568,"thread_start_us":578361,"threads_started":1,"wal-append.queue_time_us":789372}
I20260925 03:05:15.146775 27232 consensus_queue.cc:1035] T 7993f9c3e2a44f408bbf5d1e395e2306 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260925 03:05:15.134369 27267 raft_consensus.cc:482] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:15.151319 27267 raft_consensus.cc:504] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:15.104728 27403 leader_election.cc:288] T cc2df40db08649c9a75e90000b878559 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:15.153450 27267 leader_election.cc:288] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:05:15.109501 27384 raft_consensus.cc:3032] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c [term 7 LEADER]: Stepping down as leader of term 7
I20260925 03:05:15.165918 27384 raft_consensus.cc:729] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c [term 7 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:05:15.166739 27384 consensus_queue.cc:260] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:15.167936 27384 raft_consensus.cc:3037] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Advancing to term 8
I20260925 03:05:15.169939 27268 consensus_queue.cc:260] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:15.140528 24466 raft_consensus.cc:1226] T 0be4e0aa984f4f2b81a58e382e434ad5 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20260925 03:05:15.128563 24473 raft_consensus.cc:1226] T edcf6e6a422d4d198d4fbb0f7d3d845d P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 5. Current term is 6. Ops: []
I20260925 03:05:15.133136 24586 leader_election.cc:302] T 74443a0a1dc34d88a2f42e3b722db405 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:05:15.133746 24619 raft_consensus.cc:1261] T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
W20260925 03:05:15.200982 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 368666us
I20260925 03:05:15.199002 24470 raft_consensus.cc:2454] T 4e5332a5d329493991dff994e70ea92c P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 9.
I20260925 03:05:15.201962 24437 leader_election.cc:302] T 11bb46bce3564841b7674e6a693bd2e6 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:05:15.202531 24620 raft_consensus.cc:2454] T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 9.
I20260925 03:05:15.202832 27499 raft_consensus.cc:2781] T 11bb46bce3564841b7674e6a693bd2e6 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260925 03:05:15.203229 27499 raft_consensus.cc:482] T 11bb46bce3564841b7674e6a693bd2e6 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:15.203594 27499 raft_consensus.cc:3037] T 11bb46bce3564841b7674e6a693bd2e6 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Advancing to term 6
W20260925 03:05:15.213794 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 80726.2us
I20260925 03:05:15.214455 24437 leader_election.cc:302] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:05:15.175796 27254 raft_consensus.cc:504] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:15.126803 27022 raft_consensus.cc:729] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [term 8 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:05:15.218245 24468 raft_consensus.cc:1226] T c85f5d1d3f004cf5817bb594fd8e6469 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 4. Current term is 5. Ops: []
I20260925 03:05:15.219311 24616 raft_consensus.cc:1226] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 8. Current term is 9. Ops: [8.6-8.6]
I20260925 03:05:15.221902 25197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15861cd28e354598a3c6bfc4337c34bb" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43"
W20260925 03:05:15.225615 24436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 341903us
I20260925 03:05:15.229562 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c85f5d1d3f004cf5817bb594fd8e6469" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:05:15.231410 24437 leader_election.cc:302] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:15.164450 27488 raft_consensus.cc:504] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:15.235710 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c81a1b3b616492ab6c964f45af36077" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:15.237303 25196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43"
I20260925 03:05:15.240680 24547 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 1531) took 1045 ms. Trace:
I20260925 03:05:15.244791 25196 raft_consensus.cc:2379] T 7e49248643df4131a1c42e74244e05c2 P 1bc1f2e62ab44af995165bd5116a9d43 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 9: Already voted for candidate 1bc1f2e62ab44af995165bd5116a9d43 in this term.
I20260925 03:05:15.249481 24616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e8af2641e124f9589c5ccfdac785f63" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 9 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:05:15.217195 27267 raft_consensus.cc:2781] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260925 03:05:15.275923 25199 raft_consensus.cc:1226] T 29637f5a4a504ce6ba29033ccd0cd716 P 1bc1f2e62ab44af995165bd5116a9d43 [term 10 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 9. Current term is 10. Ops: []
I20260925 03:05:15.222272 27503 raft_consensus.cc:2726] T 74443a0a1dc34d88a2f42e3b722db405 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260925 03:05:15.275134 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f68804eb5a244d87bb395d50683c6012" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:05:15.223520 27510 consensus_queue.cc:1035] T 8e5819c15eac4544a57e5ee6c9d4f117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.058s
I20260925 03:05:15.272677 27479 raft_consensus.cc:482] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:15.277767 27407 raft_consensus.cc:2726] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260925 03:05:15.274307 27267 raft_consensus.cc:482] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:15.237776 27508 consensus_queue.cc:1035] T 442d4a29b1d743599f4b6b4871e26668 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.056s
I20260925 03:05:15.237326 27232 consensus_queue.cc:1046] T edcf6e6a422d4d198d4fbb0f7d3d845d P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.056s
I20260925 03:05:15.309614 27267 raft_consensus.cc:3037] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:05:15.310702 27022 consensus_queue.cc:260] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:15.334339 24468 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:55552 (request call id 454) took 1783 ms. Trace:
I20260925 03:05:15.335445 24468 rpcz_store.cc:276] 0925 03:05:13.543109 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:05:15.233686 (+1690577us) service_pool.cc:227] Handling call
0925 03:05:15.326879 (+ 93193us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 03:05:15.308235 27241 consensus_queue.cc:1046] T c85f5d1d3f004cf5817bb594fd8e6469 P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.023s
I20260925 03:05:15.309690 27503 consensus_queue.cc:1046] T 1414dcfe53f64a78ae413531eb8ec117 P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.024s
I20260925 03:05:15.307611 27498 raft_consensus.cc:504] T 100c113a9fd84cc8adf2ec7b33ea0093 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:15.365383 27232 raft_consensus.cc:3032] T 1414dcfe53f64a78ae413531eb8ec117 P eeab93abfdb141a895002ec9f312296c [term 4 LEADER]: Stepping down as leader of term 4
I20260925 03:05:15.365861 27232 raft_consensus.cc:729] T 1414dcfe53f64a78ae413531eb8ec117 P eeab93abfdb141a895002ec9f312296c [term 4 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:05:15.366832 27232 consensus_queue.cc:260] T 1414dcfe53f64a78ae413531eb8ec117 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:05:15.368245 25163 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 131316us
I20260925 03:05:15.308367 27479 raft_consensus.cc:504] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:05:15.370873 25165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 131989us
I20260925 03:05:15.332434 26741 raft_consensus.cc:482] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:05:15.371385 25163 leader_election.cc:302] T 29637f5a4a504ce6ba29033ccd0cd716 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:15.377769 27254 leader_election.cc:288] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:15.378818 27586 raft_consensus.cc:2726] T 29637f5a4a504ce6ba29033ccd0cd716 P 1bc1f2e62ab44af995165bd5116a9d43 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260925 03:05:15.378996 27499 raft_consensus.cc:504] T 11bb46bce3564841b7674e6a693bd2e6 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:15.348281 27460 raft_consensus.cc:482] T 25102ba08e4e49388f6176badcd10a25 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:15.380877 26741 raft_consensus.cc:504] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:15.380328 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c81a1b3b616492ab6c964f45af36077" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:15.381094 27460 raft_consensus.cc:504] T 25102ba08e4e49388f6176badcd10a25 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:05:15.381423 27582 log.cc:924] Time spent T 60d45b7d467d4e1ab92367f6d7d52b15 P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.058s user 0.000s sys 0.000s
I20260925 03:05:15.383368 27460 leader_election.cc:288] T 25102ba08e4e49388f6176badcd10a25 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:15.385977 24465 raft_consensus.cc:1226] T 100c113a9fd84cc8adf2ec7b33ea0093 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 3. Current term is 4. Ops: []
I20260925 03:05:15.382149 24619 raft_consensus.cc:2396] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1bc1f2e62ab44af995165bd5116a9d43 for term 8 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260925 03:05:15.363315 27368 raft_consensus.cc:482] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:15.330536 27346 raft_consensus.cc:504] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:15.386970 24543 raft_consensus.cc:2454] T a4abe0ea757e4886be4945c5001e529a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 4.
W20260925 03:05:15.393141 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 143870us
I20260925 03:05:15.373561 27022 raft_consensus.cc:3037] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Advancing to term 9
I20260925 03:05:15.401635 24466 raft_consensus.cc:1226] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 8. Current term is 9. Ops: []
I20260925 03:05:15.384363 27499 leader_election.cc:288] T 11bb46bce3564841b7674e6a693bd2e6 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
W20260925 03:05:15.391204 27573 log.cc:924] Time spent T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.364s user 0.008s sys 0.004s
I20260925 03:05:15.408212 24469 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1513) took 1039 ms. Trace:
W20260925 03:05:15.409333 27587 log.cc:924] Time spent T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.074s user 0.000s sys 0.000s
I20260925 03:05:15.408643 24469 rpcz_store.cc:276] 0925 03:05:14.369196 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:05:15.038650 (+669454us) service_pool.cc:227] Handling call
0925 03:05:15.047832 (+ 9182us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:05:15.048042 (+ 210us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:05:15.048065 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:05:15.066320 (+ 18255us) log.cc:842] Serialized 26 byte log entry
0925 03:05:15.347228 (+280908us) raft_consensus.cc:1631] Marking committed up to 4
0925 03:05:15.347270 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:05:15.347347 (+ 77us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:05:15.408007 (+ 60660us) raft_consensus.cc:1671] finished
0925 03:05:15.408027 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:05:15.408151 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":256108,"threads_started":1,"wal-append.queue_time_us":331866}
I20260925 03:05:15.393087 27401 raft_consensus.cc:482] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:05:15.411899 27401 raft_consensus.cc:504] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:15.414009 27401 leader_election.cc:288] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:05:15.416190 24465 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1510) took 1158 ms. Trace:
I20260925 03:05:15.416625 24465 rpcz_store.cc:276] 0925 03:05:14.257844 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:05:14.927178 (+669334us) service_pool.cc:227] Handling call
0925 03:05:14.943657 (+ 16479us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:05:15.386381 (+442724us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:05:15.415943 (+ 29562us) spinlock_profiling.cc:243] Waited 442 ms on lock 0x7b54003ec6d4. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d0f51 00007f43764cf99e 00007f437b4112e9 00007f4374af7684 00007f4374af7612 00007f4374af7541 00007f4374af74bc 00007f4374af6772 00007f43748fa94c 00007f43748f9d86
0925 03:05:15.416130 (+ 187us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1016802688}
I20260925 03:05:15.418004 27346 leader_election.cc:288] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 9 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:15.418432 26741 leader_election.cc:288] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
W20260925 03:05:15.438102 27588 log.cc:924] Time spent T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.097s user 0.002s sys 0.002s
I20260925 03:05:15.392865 27368 raft_consensus.cc:504] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:15.438984 27268 raft_consensus.cc:3037] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Advancing to term 9
I20260925 03:05:15.439484 24621 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38570 (request call id 1729) took 1075 ms. Trace:
W20260925 03:05:15.440804 27584 log.cc:924] Time spent T e6b6ec010caf4689b8c90449764f5cf9 P 869b0a8575b54d08ae2817fad5dd42d5: Append to log took a long time: real 0.109s user 0.001s sys 0.000s
I20260925 03:05:15.441573 27230 raft_consensus.cc:3032] T c85f5d1d3f004cf5817bb594fd8e6469 P eeab93abfdb141a895002ec9f312296c [term 4 LEADER]: Stepping down as leader of term 4
I20260925 03:05:15.442015 27230 raft_consensus.cc:729] T c85f5d1d3f004cf5817bb594fd8e6469 P eeab93abfdb141a895002ec9f312296c [term 4 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:05:15.442870 27230 consensus_queue.cc:260] T c85f5d1d3f004cf5817bb594fd8e6469 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:15.444049 27230 raft_consensus.cc:3037] T c85f5d1d3f004cf5817bb594fd8e6469 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:05:15.445277 27385 raft_consensus.cc:504] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:15.446311 24613 raft_consensus.cc:1226] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [term 11 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 10. Current term is 11. Ops: []
I20260925 03:05:15.448814 24470 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 1514) took 1079 ms. Trace:
I20260925 03:05:15.449189 24470 rpcz_store.cc:276] 0925 03:05:14.369521 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:05:14.534413 (+164892us) service_pool.cc:227] Handling call
0925 03:05:15.448613 (+914200us) spinlock_profiling.cc:243] Waited 281 ms on lock 0x7b54003eb2d4. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d42ae 00007f437b411c27 00007f4374af9c04 00007f4374af9b92 00007f4374af9ac1 00007f4374af9a3c 00007f4374af8cf2 00007f43748fa94c 00007f43748f9d86 00007f43748f972f
0925 03:05:15.448734 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":646537088}
I20260925 03:05:15.449908 27498 leader_election.cc:288] T 100c113a9fd84cc8adf2ec7b33ea0093 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:15.450491 27414 raft_consensus.cc:504] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:15.241117 24547 rpcz_store.cc:276] 0925 03:05:14.195430 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:05:14.336487 (+141057us) service_pool.cc:227] Handling call
0925 03:05:14.366004 (+ 29517us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:05:14.366289 (+ 285us) raft_consensus.cc:1520] Early marking committed up to index 4
0925 03:05:14.965098 (+598809us) log.cc:842] Serialized 14 byte log entry
0925 03:05:15.240336 (+275238us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:05:15.240388 (+ 52us) raft_consensus.cc:1631] Marking committed up to 4
0925 03:05:15.240435 (+ 47us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:05:15.240498 (+ 63us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:05:15.240623 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":731977,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":41,"raft.queue_time_us":7582,"thread_start_us":844952,"threads_started":2}
I20260925 03:05:15.439829 24621 rpcz_store.cc:276] 0925 03:05:14.363971 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:05:14.370301 (+ 6330us) service_pool.cc:227] Handling call
0925 03:05:14.370659 (+ 358us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:05:14.371019 (+ 360us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:05:14.371046 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:05:14.371501 (+ 455us) log.cc:842] Serialized 26 byte log entry
0925 03:05:14.935492 (+563991us) raft_consensus.cc:1631] Marking committed up to 5
0925 03:05:14.935536 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:05:14.936400 (+ 864us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:05:15.372187 (+435787us) spinlock_profiling.cc:243] Waited 1.15 ms on lock 0x7b24003e5d78. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f43733ecf11 00007f4373400cf4 00007f43733fedfb 00007f437348d559 00007f437348d1e5 00007f437348c82d 00007f437613b46e 00007f4376152c0c 00007f4376151f36 00007f4376151b58 00007f437615b551
0925 03:05:15.439156 (+ 66969us) raft_consensus.cc:1671] finished
0925 03:05:15.439180 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:05:15.439408 (+ 228us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":708,"raft.run_cpu_time_us":77,"raft.run_wall_time_us":78,"spinlock_wait_cycles":2637696,"thread_start_us":563633,"threads_started":1,"wal-append.queue_time_us":655186}
I20260925 03:05:15.455204 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25102ba08e4e49388f6176badcd10a25" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:15.456842 25163 leader_election.cc:302] T a4abe0ea757e4886be4945c5001e529a P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
W20260925 03:05:15.457129 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 183051us
I20260925 03:05:15.385632 27262 consensus_queue.cc:1035] T a7aa47c48404465b94a36fd819fe3c7a P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.130s
I20260925 03:05:15.445152 27328 raft_consensus.cc:3032] T edcf6e6a422d4d198d4fbb0f7d3d845d P eeab93abfdb141a895002ec9f312296c [term 5 LEADER]: Stepping down as leader of term 5
I20260925 03:05:15.461213 24615 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38570 (request call id 1683) took 3488 ms. Trace:
I20260925 03:05:15.448662 27488 leader_election.cc:288] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 9 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:05:15.467453 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9c33fd100ff44c1a3247319a324a044" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:05:15.470808 24619 raft_consensus.cc:3037] T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Advancing to term 10
W20260925 03:05:15.459175 27577 log.cc:924] Time spent T abd5b8dc1b7e49e3aa9af900d73ff0e7 P 869b0a8575b54d08ae2817fad5dd42d5: Append to log took a long time: real 0.163s user 0.000s sys 0.005s
W20260925 03:05:15.458475 27579 log.cc:924] Time spent T 11bb46bce3564841b7674e6a693bd2e6 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.153s user 0.003s sys 0.000s
I20260925 03:05:15.452589 27414 leader_election.cc:288] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 10 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:15.466524 27328 raft_consensus.cc:729] T edcf6e6a422d4d198d4fbb0f7d3d845d P eeab93abfdb141a895002ec9f312296c [term 5 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:05:15.473912 25196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c81a1b3b616492ab6c964f45af36077" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:05:15.477193 25196 raft_consensus.cc:2454] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 7.
I20260925 03:05:15.464463 24615 rpcz_store.cc:276] 0925 03:05:11.970144 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:05:12.593325 (+623181us) service_pool.cc:227] Handling call
0925 03:05:12.593623 (+ 298us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:05:14.284018 (+1690395us) raft_consensus.cc:1520] Early marking committed up to index 5
0925 03:05:14.794877 (+510859us) log.cc:842] Serialized 14 byte log entry
0925 03:05:15.202197 (+407320us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 03:05:15.202244 (+ 47us) raft_consensus.cc:1631] Marking committed up to 5
0925 03:05:15.202284 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:05:15.458133 (+255849us) spinlock_profiling.cc:243] Waited 1.65 s on lock 0x7b54000320d4. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d0f51 00007f43764cf99e 00007f437b4112e9 00007f4374af7684 00007f4374af7612 00007f4374af7541 00007f4374af74bc 00007f4374af6772 00007f43748fa94c 00007f43748f9d86
0925 03:05:15.458175 (+ 42us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:05:15.458301 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":64294,"prepare.queue_time_us":77346,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":39,"raft.queue_time_us":51354,"raft.run_cpu_time_us":276,"raft.run_wall_time_us":56283,"spinlock_wait_cycles":3788637312,"thread_start_us":748262,"threads_started":2,"wal-append.queue_time_us":483676}
I20260925 03:05:15.403519 27543 consensus_queue.cc:1046] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.7, Next index: 8, Last known committed idx: 4, Time since last communication: 0.069s
I20260925 03:05:15.481241 25191 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48424 (request call id 288) took 1273 ms. Trace:
I20260925 03:05:15.481520 25198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c81a1b3b616492ab6c964f45af36077" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:05:15.482232 27328 consensus_queue.cc:260] T edcf6e6a422d4d198d4fbb0f7d3d845d P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:15.483659 27328 raft_consensus.cc:3037] T edcf6e6a422d4d198d4fbb0f7d3d845d P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:05:15.493448 27385 leader_election.cc:288] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 11 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:05:15.493860 27510 raft_consensus.cc:482] T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:15.494378 27510 raft_consensus.cc:504] T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:05:15.495682 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 264357us
W20260925 03:05:15.496078 24436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 256122us
W20260925 03:05:15.496608 27583 log.cc:924] Time spent T 293ef69caf514ac98510a68495f54eda P 869b0a8575b54d08ae2817fad5dd42d5: Append to log took a long time: real 0.152s user 0.000s sys 0.003s
I20260925 03:05:15.467005 27533 raft_consensus.cc:2781] T a4abe0ea757e4886be4945c5001e529a P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 FOLLOWER]: Leader election won for term 4
I20260925 03:05:15.498076 27533 raft_consensus.cc:686] T a4abe0ea757e4886be4945c5001e529a P 1bc1f2e62ab44af995165bd5116a9d43 [term 4 LEADER]: Becoming Leader. State: Replica: 1bc1f2e62ab44af995165bd5116a9d43, State: Running, Role: LEADER
I20260925 03:05:15.499071 27533 consensus_queue.cc:237] T a4abe0ea757e4886be4945c5001e529a P 1bc1f2e62ab44af995165bd5116a9d43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:15.499768 27589 consensus_queue.cc:1046] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: INVALID_TERM, Last received: 10.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260925 03:05:15.501469 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35992316cb8d4c028333388eff02ecde" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:15.502084 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c81a1b3b616492ab6c964f45af36077" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:15.502254 24465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c81a1b3b616492ab6c964f45af36077" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:15.496454 24437 leader_election.cc:302] T 7b8e0d260a884ecdb20f79de46c36b19 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:15.482278 25198 raft_consensus.cc:2379] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c in current term 7: Already voted for candidate 1bc1f2e62ab44af995165bd5116a9d43 in this term.
I20260925 03:05:15.496170 27363 raft_consensus.cc:2726] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260925 03:05:15.501467 27555 raft_consensus.cc:3032] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [term 10 LEADER]: Stepping down as leader of term 10
I20260925 03:05:15.502485 27508 raft_consensus.cc:482] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:15.532963 24615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74443a0a1dc34d88a2f42e3b722db405" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
W20260925 03:05:15.493322 27585 log.cc:924] Time spent T 8e5819c15eac4544a57e5ee6c9d4f117 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.131s user 0.003s sys 0.000s
I20260925 03:05:15.536984 24469 raft_consensus.cc:2396] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1bc1f2e62ab44af995165bd5116a9d43 for term 8 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260925 03:05:15.532733 25197 raft_consensus.cc:2454] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 7.
I20260925 03:05:15.539211 27590 raft_consensus.cc:3037] T 7b8e0d260a884ecdb20f79de46c36b19 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Advancing to term 9
I20260925 03:05:15.539598 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bb46bce3564841b7674e6a693bd2e6" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:05:15.539994 24437 leader_election.cc:302] T ceab6b0fccd54e909a64f66fe81a9a1f P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:15.540128 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b030d84c670b48d5bff337ccc0c77626" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:15.554986 24615 raft_consensus.cc:2454] T 74443a0a1dc34d88a2f42e3b722db405 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 7.
I20260925 03:05:15.541218 27267 raft_consensus.cc:504] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:15.542407 24539 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36786 (request call id 1529) took 1353 ms. Trace:
I20260925 03:05:15.542840 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7691c6b475428c82d6181d693d26f3" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:05:15.543241 24618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "100c113a9fd84cc8adf2ec7b33ea0093" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:05:15.543633 27479 leader_election.cc:288] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:15.497653 27427 raft_consensus.cc:482] T c88cbd39e01f44628a094af516be2833 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:05:15.544874 27368 leader_election.cc:288] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:15.545755 24464 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1512) took 1255 ms. Trace:
I20260925 03:05:15.534493 24436 leader_election.cc:302] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:05:15.551340 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f1fac028a144058b92331c0d9b44ebf" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:15.555928 24588 leader_election.cc:302] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:05:15.557220 24619 raft_consensus.cc:2454] T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 10.
I20260925 03:05:15.496408 27510 leader_election.cc:288] T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:15.573932 25165 leader_election.cc:302] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
W20260925 03:05:15.566535 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 87595.5us
I20260925 03:05:15.547000 27503 raft_consensus.cc:482] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:15.576898 27503 raft_consensus.cc:504] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:15.578912 27503 leader_election.cc:288] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:05:15.567330 27508 raft_consensus.cc:504] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:15.581873 27508 leader_election.cc:288] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:05:15.582568 27363 raft_consensus.cc:2726] T ceab6b0fccd54e909a64f66fe81a9a1f P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260925 03:05:15.584450 27267 leader_election.cc:288] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:15.585147 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4abe0ea757e4886be4945c5001e529a" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
W20260925 03:05:15.585978 27593 log.cc:924] Time spent T 48ca6ccd98194cf2af7b5278d15bf4ea P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.062s user 0.000s sys 0.004s
I20260925 03:05:15.588217 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e725b82985746b39c0cc8f671c95353" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:05:15.598120 24545 raft_consensus.cc:2362] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 8. Re-sending same reply.
I20260925 03:05:15.585999 24548 raft_consensus.cc:2379] T a4abe0ea757e4886be4945c5001e529a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c in current term 4: Already voted for candidate 1bc1f2e62ab44af995165bd5116a9d43 in this term.
W20260925 03:05:15.606079 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 66159.6us
I20260925 03:05:15.568679 24464 rpcz_store.cc:276] 0925 03:05:14.289941 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:05:15.020379 (+730438us) service_pool.cc:227] Handling call
0925 03:05:15.020653 (+ 274us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:05:15.024914 (+ 4261us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:05:15.024943 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:05:15.046501 (+ 21558us) log.cc:842] Serialized 26 byte log entry
0925 03:05:15.321987 (+275486us) raft_consensus.cc:1631] Marking committed up to 4
0925 03:05:15.322024 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:05:15.322095 (+ 71us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:05:15.545514 (+223419us) raft_consensus.cc:1671] finished
0925 03:05:15.545539 (+ 25us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:05:15.545684 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":84,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":4961,"thread_start_us":254003,"threads_started":1,"wal-append.queue_time_us":329931}
I20260925 03:05:15.492453 27488 consensus_queue.cc:1046] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.067s
I20260925 03:05:15.638768 25197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e92f377a79354a6ebb4eec67303c13e3" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:05:15.639063 24588 leader_election.cc:302] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:05:15.640936 27252 raft_consensus.cc:3037] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:05:15.560894 24539 rpcz_store.cc:276] 0925 03:05:14.188955 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:05:14.327341 (+138386us) service_pool.cc:227] Handling call
0925 03:05:14.327667 (+ 326us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:05:14.361979 (+ 34312us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:05:14.362010 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:05:14.447985 (+ 85975us) log.cc:842] Serialized 26 byte log entry
0925 03:05:14.987440 (+539455us) raft_consensus.cc:1631] Marking committed up to 2
0925 03:05:14.987477 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:05:14.987546 (+ 69us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:05:15.542201 (+554655us) raft_consensus.cc:1671] finished
0925 03:05:15.542221 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:05:15.542349 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":21848,"raft.queue_time_us":10826,"raft.run_cpu_time_us":138,"raft.run_wall_time_us":31428,"thread_start_us":493173,"threads_started":1,"wal-append.queue_time_us":786107}
I20260925 03:05:15.591457 24467 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1517) took 1185 ms. Trace:
I20260925 03:05:15.643795 24511 leader_election.cc:302] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:05:15.607026 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f1fac028a144058b92331c0d9b44ebf" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:05:15.606958 24437 leader_election.cc:302] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:05:15.614643 25198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:05:15.645547 25198 raft_consensus.cc:2379] T 7e49248643df4131a1c42e74244e05c2 P 1bc1f2e62ab44af995165bd5116a9d43 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 in current term 9: Already voted for candidate 1bc1f2e62ab44af995165bd5116a9d43 in this term.
I20260925 03:05:15.615329 24616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5278a60f97da4e10aecd50a8d9a9a43c" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:05:15.615761 24540 raft_consensus.cc:1226] T 100c113a9fd84cc8adf2ec7b33ea0093 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 3. Current term is 4. Ops: []
I20260925 03:05:15.554934 27555 raft_consensus.cc:729] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:05:15.650600 24620 pending_rounds.cc:76] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c: Aborting all ops after (but not including) 5
I20260925 03:05:15.650976 24620 pending_rounds.cc:97] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c: Aborting uncommitted NO_OP operation due to leader change: 8.6
I20260925 03:05:15.651230 24620 raft_consensus.cc:2866] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260925 03:05:15.596901 26741 raft_consensus.cc:2781] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Leader election won for term 9
I20260925 03:05:15.506063 25191 rpcz_store.cc:276] 0925 03:05:14.204932 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:05:14.207937 (+ 3005us) service_pool.cc:227] Handling call
0925 03:05:14.222661 (+ 14724us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 03:05:15.026790 (+804129us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:05:15.469813 (+443023us) spinlock_profiling.cc:243] Waited 754 ms on lock 0x7b5400641754. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d0f51 00007f43764cf99e 00007f437b4112e9 00007f4374af7684 00007f4374af7612 00007f4374af7541 00007f4374af74bc 00007f4374af6772 00007f43748fa94c 00007f43748f9d86
0925 03:05:15.478009 (+ 8196us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1733465472}
I20260925 03:05:15.565634 26762 consensus_queue.cc:1046] T 100c113a9fd84cc8adf2ec7b33ea0093 P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.061s
I20260925 03:05:15.630687 27232 raft_consensus.cc:3037] T 1414dcfe53f64a78ae413531eb8ec117 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:05:15.575943 27427 raft_consensus.cc:504] T c88cbd39e01f44628a094af516be2833 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:15.631536 27339 raft_consensus.cc:482] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:15.696915 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e92f377a79354a6ebb4eec67303c13e3" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:15.598951 27609 raft_consensus.cc:2726] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260925 03:05:15.633556 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35992316cb8d4c028333388eff02ecde" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:15.637233 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b030d84c670b48d5bff337ccc0c77626" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:05:15.597124 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:15.700040 24545 raft_consensus.cc:3037] T b030d84c670b48d5bff337ccc0c77626 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 10 FOLLOWER]: Advancing to term 11
I20260925 03:05:15.619771 27605 raft_consensus.cc:482] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:15.639463 25197 raft_consensus.cc:2454] T e92f377a79354a6ebb4eec67303c13e3 P 1bc1f2e62ab44af995165bd5116a9d43 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 7.
I20260925 03:05:15.647620 24471 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55562 (request call id 1508) took 1469 ms. Trace:
I20260925 03:05:15.649794 24548 raft_consensus.cc:2379] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c in current term 9: Already voted for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in this term.
I20260925 03:05:15.665333 27040 raft_consensus.cc:3032] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [term 8 LEADER]: Stepping down as leader of term 8
I20260925 03:05:15.665928 24586 leader_election.cc:302] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:05:15.672799 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5f9b503b0ab43999d582015a367441e" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:15.677321 24616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60d45b7d467d4e1ab92367f6d7d52b15" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 11 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:15.702813 24622 raft_consensus.cc:2379] T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 in current term 9: Already voted for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in this term.
I20260925 03:05:15.703184 24469 raft_consensus.cc:1226] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 6. Current term is 7. Ops: []
W20260925 03:05:15.678573 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 71716.8us
I20260925 03:05:15.703569 24585 leader_election.cc:302] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:05:15.703936 24437 leader_election.cc:302] T ac9357d144aa49789faaf31fe5e6bf89 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:15.680123 27555 consensus_queue.cc:260] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.5, Last appended by leader: 5, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:15.681356 24614 raft_consensus.cc:1226] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 8. Current term is 9. Ops: []
I20260925 03:05:15.678320 27451 raft_consensus.cc:2781] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260925 03:05:15.642987 24467 rpcz_store.cc:276] 0925 03:05:14.405681 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:05:15.057411 (+651730us) service_pool.cc:227] Handling call
0925 03:05:15.057717 (+ 306us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:05:15.068484 (+ 10767us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:05:15.068520 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:05:15.093732 (+ 25212us) log.cc:842] Serialized 26 byte log entry
0925 03:05:15.347682 (+253950us) raft_consensus.cc:1631] Marking committed up to 4
0925 03:05:15.347717 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:05:15.347781 (+ 64us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:05:15.591256 (+243475us) raft_consensus.cc:1671] finished
0925 03:05:15.591276 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:05:15.591396 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":80,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":12421,"thread_start_us":226987,"threads_started":1,"wal-append.queue_time_us":290780}
I20260925 03:05:15.592619 27499 raft_consensus.cc:2781] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260925 03:05:15.695125 24511 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.4:43979, user_credentials={real_user=slave}} blocked reactor thread for 51439.3us
I20260925 03:05:15.709933 24511 leader_election.cc:302] T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:05:15.626097 27368 raft_consensus.cc:3032] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 LEADER]: Stepping down as leader of term 9
I20260925 03:05:15.715195 24511 leader_election.cc:302] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:05:15.695312 24617 raft_consensus.cc:2454] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 8.
I20260925 03:05:15.699705 24547 raft_consensus.cc:2454] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 9.
I20260925 03:05:15.708036 24619 raft_consensus.cc:2454] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 10.
I20260925 03:05:15.700764 24471 rpcz_store.cc:276] 0925 03:05:14.178135 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:05:14.243377 (+ 65242us) service_pool.cc:227] Handling call
0925 03:05:14.243700 (+ 323us) raft_consensus.cc:1457] Updating replica for 1 ops
0925 03:05:14.283358 (+ 39658us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 03:05:14.283392 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0925 03:05:14.351473 (+ 68081us) log.cc:842] Serialized 26 byte log entry
0925 03:05:14.980849 (+629376us) raft_consensus.cc:1631] Marking committed up to 4
0925 03:05:14.980884 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 03:05:14.981101 (+ 217us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0925 03:05:15.647411 (+666310us) raft_consensus.cc:1671] finished
0925 03:05:15.647432 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 03:05:15.647558 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":34100,"raft.queue_time_us":18996,"raft.run_cpu_time_us":138,"raft.run_wall_time_us":45806,"thread_start_us":529964,"threads_started":1,"wal-append.queue_time_us":797856}
I20260925 03:05:15.694116 26741 raft_consensus.cc:686] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c [term 9 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:05:15.713011 27368 raft_consensus.cc:729] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:05:15.720259 27613 raft_consensus.cc:2726] T ac9357d144aa49789faaf31fe5e6bf89 P 869b0a8575b54d08ae2817fad5dd42d5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260925 03:05:15.720747 27608 log.cc:924] Time spent T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.071s user 0.000s sys 0.000s
I20260925 03:05:15.700496 27268 raft_consensus.cc:2781] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Leader election won for term 7
W20260925 03:05:15.724918 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 59068.1us
I20260925 03:05:15.737031 24586 leader_election.cc:302] T a4abe0ea757e4886be4945c5001e529a P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 1bc1f2e62ab44af995165bd5116a9d43, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:05:15.723883 24436 leader_election.cc:302] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: 1bc1f2e62ab44af995165bd5116a9d43, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:15.731195 24545 raft_consensus.cc:2454] T b030d84c670b48d5bff337ccc0c77626 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 11.
I20260925 03:05:15.702066 27040 raft_consensus.cc:729] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [term 8 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:05:15.706161 27451 raft_consensus.cc:482] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:15.709143 27499 raft_consensus.cc:482] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:05:15.665428 27385 consensus_queue.cc:1046] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.057s
I20260925 03:05:15.720263 27414 raft_consensus.cc:2781] T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260925 03:05:15.757753 24437 leader_election.cc:302] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:15.759428 24437 leader_election.cc:302] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:05:15.733290 27488 raft_consensus.cc:2781] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260925 03:05:15.732940 27536 raft_consensus.cc:2781] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Leader election won for term 8
I20260925 03:05:15.742432 27543 raft_consensus.cc:2781] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 10 FOLLOWER]: Leader election won for term 10
I20260925 03:05:15.706468 27339 raft_consensus.cc:504] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:15.754539 27499 raft_consensus.cc:3037] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Advancing to term 8
I20260925 03:05:15.750517 27241 raft_consensus.cc:2726] T a4abe0ea757e4886be4945c5001e529a P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260925 03:05:15.746582 27555 raft_consensus.cc:3037] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [term 10 FOLLOWER]: Advancing to term 11
I20260925 03:05:15.733493 27022 consensus_queue.cc:1046] T 5278a60f97da4e10aecd50a8d9a9a43c P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.056s
I20260925 03:05:15.754220 27451 raft_consensus.cc:3037] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:05:15.753684 27508 raft_consensus.cc:3037] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Advancing to term 9
W20260925 03:05:15.789413 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 46989.2us
I20260925 03:05:15.750248 27268 raft_consensus.cc:686] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 LEADER]: Becoming Leader. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
W20260925 03:05:15.799059 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 62921.6us
I20260925 03:05:15.757477 27414 raft_consensus.cc:482] T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260925 03:05:15.792423 24511 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 77305.1us
I20260925 03:05:15.705832 27605 raft_consensus.cc:504] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:15.740232 27363 consensus_queue.cc:1046] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.057s
I20260925 03:05:15.783253 27346 raft_consensus.cc:2726] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260925 03:05:15.789672 27488 raft_consensus.cc:482] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:05:15.797961 27513 raft_consensus.cc:2781] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260925 03:05:15.817507 27414 raft_consensus.cc:3037] T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Advancing to term 8
I20260925 03:05:15.808239 27536 raft_consensus.cc:686] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [term 8 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:05:15.826923 27513 raft_consensus.cc:482] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:15.845755 27513 raft_consensus.cc:3037] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Advancing to term 10
I20260925 03:05:15.854702 27363 raft_consensus.cc:3032] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 LEADER]: Stepping down as leader of term 8
I20260925 03:05:15.856103 27513 raft_consensus.cc:504] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:15.826471 27488 raft_consensus.cc:3037] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Advancing to term 8
W20260925 03:05:15.868880 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 96040.8us
I20260925 03:05:15.846918 27268 consensus_queue.cc:237] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
W20260925 03:05:15.877032 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 117602us
I20260925 03:05:15.878754 24437 leader_election.cc:302] T a7aa47c48404465b94a36fd819fe3c7a P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
W20260925 03:05:15.901019 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 73087.4us
I20260925 03:05:15.804798 27368 consensus_queue.cc:260] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:15.866021 27363 raft_consensus.cc:729] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
W20260925 03:05:15.917675 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 106290us
I20260925 03:05:15.868958 27613 raft_consensus.cc:482] T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:15.922391 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c88cbd39e01f44628a094af516be2833" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:15.918522 27427 leader_election.cc:288] T c88cbd39e01f44628a094af516be2833 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
W20260925 03:05:15.951597 27612 log.cc:924] Time spent T 8e5819c15eac4544a57e5ee6c9d4f117 P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.075s user 0.001s sys 0.000s
I20260925 03:05:15.924856 27613 raft_consensus.cc:504] T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:05:15.952234 27611 log.cc:924] Time spent T 25102ba08e4e49388f6176badcd10a25 P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.075s user 0.001s sys 0.000s
W20260925 03:05:15.952948 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 51206us
I20260925 03:05:15.825577 26741 consensus_queue.cc:237] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:15.977550 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "293ef69caf514ac98510a68495f54eda" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:15.978211 24471 raft_consensus.cc:2454] T 293ef69caf514ac98510a68495f54eda P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 7.
I20260925 03:05:15.979888 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "293ef69caf514ac98510a68495f54eda" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:15.980531 24546 raft_consensus.cc:2454] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 8.
I20260925 03:05:15.981673 27241 consensus_queue.cc:1046] T 100c113a9fd84cc8adf2ec7b33ea0093 P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260925 03:05:15.985770 27613 leader_election.cc:288] T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
W20260925 03:05:15.987656 27617 log.cc:924] Time spent T abd5b8dc1b7e49e3aa9af900d73ff0e7 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.206s user 0.001s sys 0.000s
W20260925 03:05:15.988740 27614 log.cc:924] Time spent T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5: Append to log took a long time: real 0.070s user 0.000s sys 0.001s
I20260925 03:05:15.982571 27339 leader_election.cc:288] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:05:15.989910 25163 leader_election.cc:302] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:05:15.959982 27628 raft_consensus.cc:2726] T a7aa47c48404465b94a36fd819fe3c7a P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260925 03:05:15.989995 27252 raft_consensus.cc:2726] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260925 03:05:15.969743 27543 raft_consensus.cc:686] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 10 LEADER]: Becoming Leader. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
W20260925 03:05:15.982394 27607 log.cc:924] Time spent T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.230s user 0.000s sys 0.005s
W20260925 03:05:16.004323 24511 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 181215us
W20260925 03:05:16.076298 24514 leader_election.cc:334] T 48ca6ccd98194cf2af7b5278d15bf4ea P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187): Timed out: RequestConsensusVote RPC to 127.22.193.1:42187 timed out after 1.667s (SENT)
I20260925 03:05:16.077440 24514 leader_election.cc:302] T 48ca6ccd98194cf2af7b5278d15bf4ea P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:16.078367 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48ca6ccd98194cf2af7b5278d15bf4ea" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:16.002449 27254 raft_consensus.cc:2781] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260925 03:05:16.002249 27590 raft_consensus.cc:2726] T 7b8e0d260a884ecdb20f79de46c36b19 P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260925 03:05:16.113448 27053 raft_consensus.cc:3032] T 100c113a9fd84cc8adf2ec7b33ea0093 P eeab93abfdb141a895002ec9f312296c [term 3 LEADER]: Stepping down as leader of term 3
I20260925 03:05:16.113883 27053 raft_consensus.cc:729] T 100c113a9fd84cc8adf2ec7b33ea0093 P eeab93abfdb141a895002ec9f312296c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:05:16.115221 27053 consensus_queue.cc:260] T 100c113a9fd84cc8adf2ec7b33ea0093 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:16.116609 27053 raft_consensus.cc:3037] T 100c113a9fd84cc8adf2ec7b33ea0093 P eeab93abfdb141a895002ec9f312296c [term 3 FOLLOWER]: Advancing to term 4
I20260925 03:05:15.975493 27363 consensus_queue.cc:260] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:16.113754 27499 raft_consensus.cc:504] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:16.121361 27363 raft_consensus.cc:3037] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Advancing to term 9
W20260925 03:05:16.121474 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 45128.3us
W20260925 03:05:16.084197 27616 log.cc:924] Time spent T a4abe0ea757e4886be4945c5001e529a P 1bc1f2e62ab44af995165bd5116a9d43: Append to log took a long time: real 0.108s user 0.000s sys 0.000s
I20260925 03:05:16.105288 27629 raft_consensus.cc:2726] T 48ca6ccd98194cf2af7b5278d15bf4ea P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260925 03:05:16.105844 27328 consensus_meta.cc:287] T edcf6e6a422d4d198d4fbb0f7d3d845d P eeab93abfdb141a895002ec9f312296c: Time spent flushing consensus metadata: real 0.622s user 0.013s sys 0.000s
I20260925 03:05:16.108574 24469 raft_consensus.cc:1261] T 06f4a7eaf6534e4986c9fd35911038e4 P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260925 03:05:15.974342 27536 consensus_queue.cc:237] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:16.125459 23300 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260925 03:05:16.127128 27605 leader_election.cc:288] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:05:16.077062 27513 leader_election.cc:288] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 10 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:16.112360 26762 raft_consensus.cc:3032] T 5278a60f97da4e10aecd50a8d9a9a43c P eeab93abfdb141a895002ec9f312296c [term 6 LEADER]: Stepping down as leader of term 6
I20260925 03:05:16.109174 27414 raft_consensus.cc:504] T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:16.129881 26762 raft_consensus.cc:729] T 5278a60f97da4e10aecd50a8d9a9a43c P eeab93abfdb141a895002ec9f312296c [term 6 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:05:16.130894 26762 consensus_queue.cc:260] T 5278a60f97da4e10aecd50a8d9a9a43c P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:16.083282 27254 raft_consensus.cc:482] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:05:16.153189 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c23337a7cb74184a0e57b7095cb6a11" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:05:16.122934 27499 leader_election.cc:288] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 8 election: Requested vote from peers eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
W20260925 03:05:16.110271 27610 log.cc:924] Time spent T 00000000000000000000000000000000 P 2d92cc0732c4490eb7d98d353826cce1: Append to log took a long time: real 0.431s user 0.005s sys 0.000s
I20260925 03:05:16.131300 27414 leader_election.cc:288] T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 8 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:16.138574 24511 leader_election.cc:302] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:05:16.161082 24622 raft_consensus.cc:1261] T a4abe0ea757e4886be4945c5001e529a P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Refusing update from remote peer 1bc1f2e62ab44af995165bd5116a9d43: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
W20260925 03:05:16.185544 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 207695us
W20260925 03:05:16.146612 24514 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.22.193.1:42187 timed out after 1.748s (SENT) [suppressed 18 similar messages]
I20260925 03:05:16.190034 24586 leader_election.cc:302] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:05:16.189196 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c81a1b3b616492ab6c964f45af36077" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:05:16.191829 24619 raft_consensus.cc:3037] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Advancing to term 8
I20260925 03:05:16.157727 27254 raft_consensus.cc:3037] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43 [term 8 FOLLOWER]: Advancing to term 9
I20260925 03:05:16.200964 27533 consensus_queue.cc:1035] T a4abe0ea757e4886be4945c5001e529a P 1bc1f2e62ab44af995165bd5116a9d43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260925 03:05:16.184976 27631 raft_consensus.cc:2781] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260925 03:05:16.205929 27631 raft_consensus.cc:482] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:05:16.206344 27631 raft_consensus.cc:3037] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 10 FOLLOWER]: Advancing to term 11
W20260925 03:05:16.206808 27623 log.cc:924] Time spent T 25102ba08e4e49388f6176badcd10a25 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.322s user 0.000s sys 0.002s
I20260925 03:05:16.209188 27040 consensus_queue.cc:260] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:16.210269 27040 raft_consensus.cc:3037] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Advancing to term 9
I20260925 03:05:16.213663 27508 raft_consensus.cc:2726] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260925 03:05:16.224284 24586 leader_election.cc:302] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:05:16.228806 24540 raft_consensus.cc:1261] T a4abe0ea757e4886be4945c5001e529a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Refusing update from remote peer 1bc1f2e62ab44af995165bd5116a9d43: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260925 03:05:16.232191 24619 raft_consensus.cc:2454] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 8.
I20260925 03:05:16.193638 26762 raft_consensus.cc:3037] T 5278a60f97da4e10aecd50a8d9a9a43c P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:05:16.223901 25191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c81a1b3b616492ab6c964f45af36077" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43"
W20260925 03:05:16.223543 27407 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.552s user 0.007s sys 0.017s
I20260925 03:05:16.248659 24436 leader_election.cc:302] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:05:16.228096 27252 raft_consensus.cc:2726] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260925 03:05:16.193495 27488 raft_consensus.cc:504] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:16.254630 24586 leader_election.cc:302] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:05:16.257679 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceab6b0fccd54e909a64f66fe81a9a1f" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:16.258490 24471 raft_consensus.cc:2454] T ceab6b0fccd54e909a64f66fe81a9a1f P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 9.
I20260925 03:05:16.238278 27513 raft_consensus.cc:482] T 8e5819c15eac4544a57e5ee6c9d4f117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260925 03:05:16.259922 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 360202us
I20260925 03:05:16.261778 24437 leader_election.cc:302] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:16.263134 24437 leader_election.cc:302] T 3d89396576884f7c8901dba54f783947 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:05:16.238344 27414 raft_consensus.cc:482] T c85f5d1d3f004cf5817bb594fd8e6469 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:16.249086 25191 raft_consensus.cc:3037] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43 [term 7 FOLLOWER]: Advancing to term 8
I20260925 03:05:16.288941 27641 raft_consensus.cc:2781] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [term 11 FOLLOWER]: Leader election won for term 11
I20260925 03:05:16.289605 27641 raft_consensus.cc:686] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [term 11 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:05:16.290541 27641 consensus_queue.cc:237] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.5, Last appended by leader: 5, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:05:16.249150 27407 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.577s user 0.008s sys 0.018s
I20260925 03:05:16.293239 27485 consensus_queue.cc:1035] T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260925 03:05:16.230193 27543 consensus_queue.cc:237] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 8.4, Last appended by leader: 4, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:16.264932 27628 raft_consensus.cc:3037] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Advancing to term 8
I20260925 03:05:16.296795 27536 raft_consensus.cc:482] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:05:16.262852 27230 raft_consensus.cc:2781] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260925 03:05:16.297676 27230 raft_consensus.cc:482] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Starting leader election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:05:16.298067 27230 raft_consensus.cc:3037] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Advancing to term 10
I20260925 03:05:16.300401 27513 raft_consensus.cc:504] T 8e5819c15eac4544a57e5ee6c9d4f117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:16.297331 27536 raft_consensus.cc:504] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:16.302493 27513 leader_election.cc:288] T 8e5819c15eac4544a57e5ee6c9d4f117 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:16.274420 27267 raft_consensus.cc:2781] T 3d89396576884f7c8901dba54f783947 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260925 03:05:16.303181 27267 raft_consensus.cc:482] T 3d89396576884f7c8901dba54f783947 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:16.303591 24614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e5819c15eac4544a57e5ee6c9d4f117" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:16.262764 27637 raft_consensus.cc:2781] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Leader election won for term 8
I20260925 03:05:16.243403 27339 consensus_queue.cc:1035] T a4abe0ea757e4886be4945c5001e529a P 1bc1f2e62ab44af995165bd5116a9d43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.039s
I20260925 03:05:16.310879 27536 leader_election.cc:288] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
W20260925 03:05:16.313131 25162 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 114995us
I20260925 03:05:16.286067 24437 leader_election.cc:302] T 293ef69caf514ac98510a68495f54eda P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: 1bc1f2e62ab44af995165bd5116a9d43, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:05:16.285773 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74443a0a1dc34d88a2f42e3b722db405" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:05:16.316532 24622 raft_consensus.cc:3037] T 74443a0a1dc34d88a2f42e3b722db405 P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Advancing to term 8
I20260925 03:05:16.303565 27267 raft_consensus.cc:3037] T 3d89396576884f7c8901dba54f783947 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:05:16.325810 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceab6b0fccd54e909a64f66fe81a9a1f" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:16.326470 24546 raft_consensus.cc:2454] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 9.
I20260925 03:05:16.327410 24437 leader_election.cc:302] T b627ecb4a5384ef392aa695e06599c39 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:05:16.329833 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f22ddd7d1c444dc58f45a1fb0b3e02bc" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:16.330483 24545 raft_consensus.cc:2454] T f22ddd7d1c444dc58f45a1fb0b3e02bc P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 9.
I20260925 03:05:16.343428 24544 raft_consensus.cc:1226] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 8. Current term is 9. Ops: []
I20260925 03:05:16.344808 24541 raft_consensus.cc:1261] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260925 03:05:16.347734 24437 leader_election.cc:398] T 4e5332a5d329493991dff994e70ea92c P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 8 election: Vote denied by peer a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389) with higher term. Message: Invalid argument: T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 8. Current term is 9.
I20260925 03:05:16.350037 27230 raft_consensus.cc:504] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:16.351370 25191 raft_consensus.cc:2454] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 8.
I20260925 03:05:16.352308 27230 leader_election.cc:288] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 10 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:05:16.332901 27590 raft_consensus.cc:2781] T b627ecb4a5384ef392aa695e06599c39 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260925 03:05:16.324357 27637 raft_consensus.cc:686] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 LEADER]: Becoming Leader. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:05:16.333087 27508 raft_consensus.cc:3037] T 293ef69caf514ac98510a68495f54eda P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Advancing to term 8
I20260925 03:05:16.303028 27631 raft_consensus.cc:504] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:16.345728 27627 raft_consensus.cc:482] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260925 03:05:16.329764 27555 consensus_meta.cc:287] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5: Time spent flushing consensus metadata: real 0.523s user 0.003s sys 0.006s
I20260925 03:05:16.399333 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25102ba08e4e49388f6176badcd10a25" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 8 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:16.318609 27414 raft_consensus.cc:504] T c85f5d1d3f004cf5817bb594fd8e6469 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:16.366962 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcf6e6a422d4d198d4fbb0f7d3d845d" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:05:16.419873 24545 raft_consensus.cc:3037] T edcf6e6a422d4d198d4fbb0f7d3d845d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:05:16.420081 24473 raft_consensus.cc:1226] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 LEADER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 8. Current term is 9. Ops: []
I20260925 03:05:16.424217 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "293ef69caf514ac98510a68495f54eda" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 8 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:16.406260 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e8af2641e124f9589c5ccfdac785f63" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 9 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:05:16.407264 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ac285175464fbda67211d9574db415" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:05:16.428148 24547 raft_consensus.cc:2396] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 for term 9 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 5.
I20260925 03:05:16.407869 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f68804eb5a244d87bb395d50683c6012" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:05:16.428923 24541 raft_consensus.cc:2379] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 in current term 8: Already voted for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in this term.
I20260925 03:05:16.411962 24467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21c43af5e6844fdaacc8c625fa01fd8d" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:16.412866 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "550118bc1a0641ce9803bdbf43f2069f" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:16.401031 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f22ddd7d1c444dc58f45a1fb0b3e02bc" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:05:16.430852 24539 raft_consensus.cc:2396] T f22ddd7d1c444dc58f45a1fb0b3e02bc P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 for term 9 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
I20260925 03:05:16.267588 24619 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3d89396576884f7c8901dba54f783947"
dest_uuid: "eeab93abfdb141a895002ec9f312296c"
mode: GRACEFUL
new_leader_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
from {username='slave'} at 127.0.0.1:44570
I20260925 03:05:16.393375 27590 raft_consensus.cc:482] T b627ecb4a5384ef392aa695e06599c39 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Starting leader election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:05:16.332024 27536 consensus_queue.cc:1035] T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.080s
I20260925 03:05:16.318789 27451 raft_consensus.cc:504] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:16.432770 24619 raft_consensus.cc:595] T 3d89396576884f7c8901dba54f783947 P eeab93abfdb141a895002ec9f312296c [term 5 LEADER]: Received request to transfer leadership to TS a04a37fc4aa7460cbfdbeeb4e451e1f5
W20260925 03:05:16.439502 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 103628us
I20260925 03:05:16.427117 24545 raft_consensus.cc:2454] T edcf6e6a422d4d198d4fbb0f7d3d845d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 6.
I20260925 03:05:16.444216 24465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b030d84c670b48d5bff337ccc0c77626" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 11 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:16.444804 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bb46bce3564841b7674e6a693bd2e6" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:05:16.434734 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abd5b8dc1b7e49e3aa9af900d73ff0e7" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:16.443804 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b8e0d260a884ecdb20f79de46c36b19" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:16.434103 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7691c6b475428c82d6181d693d26f3" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:05:16.437177 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48ca6ccd98194cf2af7b5278d15bf4ea" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:16.446798 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c85f5d1d3f004cf5817bb594fd8e6469" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:05:16.437373 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d89396576884f7c8901dba54f783947" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:16.437314 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c88cbd39e01f44628a094af516be2833" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:16.437813 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5278a60f97da4e10aecd50a8d9a9a43c" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:16.437597 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15861cd28e354598a3c6bfc4337c34bb" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:16.452742 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "100c113a9fd84cc8adf2ec7b33ea0093" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:05:16.480211 24465 raft_consensus.cc:2396] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for term 11 because replica has last-logged OpId of term: 10 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 4.
I20260925 03:05:16.408304 27499 raft_consensus.cc:482] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:05:16.486907 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48ca6ccd98194cf2af7b5278d15bf4ea" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:16.471935 24543 raft_consensus.cc:2379] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 in current term 9: Already voted for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in this term.
I20260925 03:05:16.489158 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5278a60f97da4e10aecd50a8d9a9a43c" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:05:16.480443 24539 raft_consensus.cc:2379] T 100c113a9fd84cc8adf2ec7b33ea0093 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 in current term 4: Already voted for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in this term.
I20260925 03:05:16.480347 24471 raft_consensus.cc:2396] T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260925 03:05:16.481040 24588 leader_election.cc:302] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:05:16.482564 24541 raft_consensus.cc:3037] T c85f5d1d3f004cf5817bb594fd8e6469 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:05:16.484969 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c23337a7cb74184a0e57b7095cb6a11" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:05:16.432262 27339 raft_consensus.cc:482] T 7e49248643df4131a1c42e74244e05c2 P 1bc1f2e62ab44af995165bd5116a9d43 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:16.433250 27642 raft_consensus.cc:482] T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:16.487534 24547 raft_consensus.cc:2454] T 48ca6ccd98194cf2af7b5278d15bf4ea P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 6.
I20260925 03:05:16.432461 27640 raft_consensus.cc:482] T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260925 03:05:16.432785 27589 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.756s user 0.001s sys 0.025s
I20260925 03:05:16.500478 24545 raft_consensus.cc:1226] T b030d84c670b48d5bff337ccc0c77626 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 11 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 10. Current term is 11. Ops: [10.5-10.5]
I20260925 03:05:16.440040 27479 raft_consensus.cc:482] T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:16.441473 27590 raft_consensus.cc:3037] T b627ecb4a5384ef392aa695e06599c39 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Advancing to term 8
I20260925 03:05:16.505503 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:16.505695 24616 raft_consensus.cc:1226] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [term 10 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 9. Current term is 10. Ops: []
I20260925 03:05:16.506287 27414 leader_election.cc:288] T c85f5d1d3f004cf5817bb594fd8e6469 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:16.508888 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:16.517102 24542 raft_consensus.cc:2379] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 in current term 9: Already voted for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in this term.
I20260925 03:05:16.517747 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e5819c15eac4544a57e5ee6c9d4f117" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
W20260925 03:05:16.519600 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 76259.8us
I20260925 03:05:16.515956 24543 raft_consensus.cc:1226] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 8. Current term is 9. Ops: [8.6-8.6]
I20260925 03:05:16.419925 27627 raft_consensus.cc:504] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:16.526391 24465 raft_consensus.cc:1226] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [term 10 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 9. Current term is 10. Ops: [7.6-9.7]
I20260925 03:05:16.528371 24542 raft_consensus.cc:1261] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 9 index: 6. (term mismatch)
I20260925 03:05:16.528790 24542 pending_rounds.cc:76] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Aborting all ops after (but not including) 5
I20260925 03:05:16.528978 24542 pending_rounds.cc:97] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Aborting uncommitted NO_OP operation due to leader change: 8.6
I20260925 03:05:16.529184 24542 raft_consensus.cc:2866] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260925 03:05:16.510202 27589 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.834s user 0.001s sys 0.026s
W20260925 03:05:16.530491 27645 log.cc:924] Time spent T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.087s user 0.001s sys 0.000s
I20260925 03:05:16.531673 27625 consensus_queue.cc:1046] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260925 03:05:16.534799 24547 raft_consensus.cc:1226] T 442d4a29b1d743599f4b6b4871e26668 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 8. Current term is 9. Ops: [8.6-8.6]
I20260925 03:05:16.539147 24467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f22ddd7d1c444dc58f45a1fb0b3e02bc" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:16.539438 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35992316cb8d4c028333388eff02ecde" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:16.541838 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c85f5d1d3f004cf5817bb594fd8e6469" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:16.542618 24619 raft_consensus.cc:2454] T c85f5d1d3f004cf5817bb594fd8e6469 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 5.
I20260925 03:05:16.543416 24467 raft_consensus.cc:2379] T f22ddd7d1c444dc58f45a1fb0b3e02bc P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c in current term 9: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:05:16.548508 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5f9b503b0ab43999d582015a367441e" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:05:16.549196 24619 raft_consensus.cc:3037] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [term 10 FOLLOWER]: Advancing to term 11
I20260925 03:05:16.550622 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06f4a7eaf6534e4986c9fd35911038e4" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:16.551267 24466 raft_consensus.cc:2454] T 06f4a7eaf6534e4986c9fd35911038e4 P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 9.
I20260925 03:05:16.555653 24470 raft_consensus.cc:2454] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 9.
I20260925 03:05:16.533953 27262 raft_consensus.cc:2781] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260925 03:05:16.601820 24471 raft_consensus.cc:1226] T 11bb46bce3564841b7674e6a693bd2e6 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 5. Current term is 6. Ops: []
I20260925 03:05:16.602700 24511 leader_election.cc:302] T c85f5d1d3f004cf5817bb594fd8e6469 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:05:16.575985 24541 raft_consensus.cc:2454] T c85f5d1d3f004cf5817bb594fd8e6469 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 5.
I20260925 03:05:16.603804 25197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e92f377a79354a6ebb4eec67303c13e3" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43"
I20260925 03:05:16.604177 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7aa47c48404465b94a36fd819fe3c7a" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:16.607245 25197 raft_consensus.cc:3037] T e92f377a79354a6ebb4eec67303c13e3 P 1bc1f2e62ab44af995165bd5116a9d43 [term 7 FOLLOWER]: Advancing to term 8
I20260925 03:05:16.533782 27631 leader_election.cc:288] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 11 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:16.607216 24619 raft_consensus.cc:2454] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 11.
I20260925 03:05:16.628113 25197 raft_consensus.cc:2454] T e92f377a79354a6ebb4eec67303c13e3 P 1bc1f2e62ab44af995165bd5116a9d43 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 8.
I20260925 03:05:16.502103 27499 raft_consensus.cc:504] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:16.560364 27488 leader_election.cc:288] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 8 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:05:16.513453 27640 raft_consensus.cc:504] T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:16.647773 24585 leader_election.cc:302] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:05:16.519311 27479 raft_consensus.cc:504] T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:16.510718 27339 raft_consensus.cc:504] T 7e49248643df4131a1c42e74244e05c2 P 1bc1f2e62ab44af995165bd5116a9d43 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:16.658298 24622 raft_consensus.cc:2454] T 74443a0a1dc34d88a2f42e3b722db405 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 8.
I20260925 03:05:16.633993 27262 raft_consensus.cc:482] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:05:16.660779 24466 raft_consensus.cc:1226] T 9e8af2641e124f9589c5ccfdac785f63 P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 8. Current term is 9. Ops: [6.6-8.7]
I20260925 03:05:16.661526 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e92f377a79354a6ebb4eec67303c13e3" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:05:16.845082 24539 raft_consensus.cc:3037] T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Advancing to term 8
I20260925 03:05:16.662886 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35992316cb8d4c028333388eff02ecde" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
W20260925 03:05:16.667436 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 119312us
I20260925 03:05:16.850330 24511 leader_election.cc:302] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:05:16.851207 24511 leader_election.cc:302] T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:05:16.757133 27667 raft_consensus.cc:482] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260925 03:05:16.742326 27653 log.cc:924] Time spent T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.197s user 0.000s sys 0.000s
W20260925 03:05:16.761458 24617 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38570 (request call id 1742) took 1532 ms (client timeout 1975 ms). Trace:
I20260925 03:05:16.800446 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
W20260925 03:05:16.859731 24617 rpcz_store.cc:269] 0925 03:05:15.229163 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:05:15.237111 (+ 7948us) service_pool.cc:227] Handling call
0925 03:05:16.761222 (+1524111us) spinlock_profiling.cc:243] Waited 337 ms on lock 0x7b540018f1d4. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d42ae 00007f437b411c27 00007f4374af9c04 00007f4374af9b92 00007f4374af9ac1 00007f4374af9a3c 00007f4374af8cf2 00007f43748fa94c 00007f43748f9d86 00007f43748f972f
0925 03:05:16.761354 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":775052032}
I20260925 03:05:16.749200 27640 leader_election.cc:288] T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:05:16.746698 27339 leader_election.cc:288] T 7e49248643df4131a1c42e74244e05c2 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:16.762909 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f68804eb5a244d87bb395d50683c6012" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:16.754915 27589 raft_consensus.cc:482] T 1414dcfe53f64a78ae413531eb8ec117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:16.762324 27384 raft_consensus.cc:2781] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Leader election won for term 8
I20260925 03:05:16.750727 27479 leader_election.cc:288] T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:05:16.804425 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1414dcfe53f64a78ae413531eb8ec117" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:16.711050 27625 raft_consensus.cc:482] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1bc1f2e62ab44af995165bd5116a9d43)
W20260925 03:05:16.762015 27657 log.cc:924] Time spent T a4abe0ea757e4886be4945c5001e529a P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.260s user 0.001s sys 0.000s
I20260925 03:05:16.753773 27414 raft_consensus.cc:482] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:16.512902 27637 consensus_queue.cc:237] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:16.761600 27499 leader_election.cc:288] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:16.643394 27510 raft_consensus.cc:2741] T c85f5d1d3f004cf5817bb594fd8e6469 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260925 03:05:16.707854 27262 raft_consensus.cc:3037] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Advancing to term 10
W20260925 03:05:16.757911 27592 log.cc:924] Time spent T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.067s user 0.000s sys 0.000s
W20260925 03:05:16.712023 24585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.4:43979, user_credentials={real_user=slave}} blocked reactor thread for 64344.7us
I20260925 03:05:16.511024 27642 raft_consensus.cc:504] T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:16.710014 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f1fac028a144058b92331c0d9b44ebf" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:16.646113 27629 raft_consensus.cc:482] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:16.833307 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9c33fd100ff44c1a3247319a324a044" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
W20260925 03:05:16.712857 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 74901.9us
I20260925 03:05:16.854948 24539 raft_consensus.cc:2454] T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 8.
W20260925 03:05:16.855446 25163 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 298766us
I20260925 03:05:16.935171 24468 raft_consensus.cc:2454] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 9.
I20260925 03:05:16.860069 24471 raft_consensus.cc:2454] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 9.
W20260925 03:05:16.728916 27644 log.cc:924] Time spent T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.084s user 0.000s sys 0.005s
I20260925 03:05:16.827725 27268 raft_consensus.cc:482] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260925 03:05:16.840785 27363 consensus_meta.cc:287] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5: Time spent flushing consensus metadata: real 0.719s user 0.008s sys 0.004s
I20260925 03:05:16.863440 24622 raft_consensus.cc:1261] T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c [term 10 FOLLOWER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 10 index: 5. (index mismatch)
I20260925 03:05:16.873591 24470 raft_consensus.cc:2379] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 8: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:05:16.877365 24467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7691c6b475428c82d6181d693d26f3" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
W20260925 03:05:16.865411 27628 consensus_meta.cc:287] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5: Time spent flushing consensus metadata: real 0.568s user 0.009s sys 0.001s
W20260925 03:05:16.865584 26762 consensus_meta.cc:287] T 5278a60f97da4e10aecd50a8d9a9a43c P eeab93abfdb141a895002ec9f312296c: Time spent flushing consensus metadata: real 0.625s user 0.010s sys 0.000s
I20260925 03:05:16.879609 24616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e92f377a79354a6ebb4eec67303c13e3" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:16.859807 27538 raft_consensus.cc:482] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:16.926744 27671 raft_consensus.cc:2781] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 11 FOLLOWER]: Leader election won for term 11
I20260925 03:05:16.917881 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442d4a29b1d743599f4b6b4871e26668" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:16.871073 27385 raft_consensus.cc:482] T edcf6e6a422d4d198d4fbb0f7d3d845d P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:16.886574 27384 raft_consensus.cc:686] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [term 8 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
W20260925 03:05:16.892678 27267 consensus_meta.cc:287] T 3d89396576884f7c8901dba54f783947 P 869b0a8575b54d08ae2817fad5dd42d5: Time spent flushing consensus metadata: real 0.564s user 0.009s sys 0.000s
I20260925 03:05:16.886221 27479 raft_consensus.cc:482] T 37a157996e944db2906110d87ecb0c54 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:05:16.882678 27451 leader_election.cc:288] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:05:16.890111 27627 leader_election.cc:288] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:16.935357 24586 leader_election.cc:302] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:05:16.939877 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367277daf8d04778b3fa4ed94d71580d" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:05:16.940766 24467 raft_consensus.cc:2379] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 9: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:05:16.942263 24619 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7b8e0d260a884ecdb20f79de46c36b19"
dest_uuid: "eeab93abfdb141a895002ec9f312296c"
mode: GRACEFUL
new_leader_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
from {username='slave'} at 127.0.0.1:44570
I20260925 03:05:16.942725 24436 leader_election.cc:302] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:05:16.882800 27589 raft_consensus.cc:504] T 1414dcfe53f64a78ae413531eb8ec117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:16.943969 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5f9b503b0ab43999d582015a367441e" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:16.929323 27632 raft_consensus.cc:2781] T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Leader election won for term 8
W20260925 03:05:16.918146 24511 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 67006.8us
I20260925 03:05:17.013003 24467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "293ef69caf514ac98510a68495f54eda" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 8 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:16.950304 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be4e0aa984f4f2b81a58e382e434ad5" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
W20260925 03:05:17.022786 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 87521.6us
I20260925 03:05:17.022725 24468 raft_consensus.cc:2454] T 0be4e0aa984f4f2b81a58e382e434ad5 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 8.
W20260925 03:05:16.954095 27652 log.cc:924] Time spent T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.094s user 0.000s sys 0.000s
I20260925 03:05:16.940784 27268 raft_consensus.cc:504] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:16.935495 27640 consensus_queue.cc:1046] T 11bb46bce3564841b7674e6a693bd2e6 P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.026s
I20260925 03:05:16.940407 25191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367277daf8d04778b3fa4ed94d71580d" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43"
I20260925 03:05:16.917940 27625 raft_consensus.cc:504] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:16.940828 27629 raft_consensus.cc:504] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:16.889914 27414 raft_consensus.cc:504] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:16.960909 27538 raft_consensus.cc:504] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:05:16.968266 27658 log.cc:924] Time spent T 442d4a29b1d743599f4b6b4871e26668 P 869b0a8575b54d08ae2817fad5dd42d5: Append to log took a long time: real 0.238s user 0.001s sys 0.000s
I20260925 03:05:16.969604 24470 raft_consensus.cc:1226] T f22ddd7d1c444dc58f45a1fb0b3e02bc P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20260925 03:05:16.950343 24473 raft_consensus.cc:2396] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for term 9 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 5.
I20260925 03:05:16.974715 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9c33fd100ff44c1a3247319a324a044" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:17.007972 27267 raft_consensus.cc:504] T 3d89396576884f7c8901dba54f783947 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:16.975797 27642 leader_election.cc:288] T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
W20260925 03:05:16.949007 27651 log.cc:924] Time spent T 60d45b7d467d4e1ab92367f6d7d52b15 P 869b0a8575b54d08ae2817fad5dd42d5: Append to log took a long time: real 0.153s user 0.000s sys 0.001s
W20260925 03:05:16.993973 27040 consensus_meta.cc:287] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c: Time spent flushing consensus metadata: real 0.783s user 0.012s sys 0.000s
I20260925 03:05:16.947458 27628 raft_consensus.cc:2726] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260925 03:05:16.996417 24614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:17.050115 24614 raft_consensus.cc:2454] T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 9.
I20260925 03:05:17.051209 24469 raft_consensus.cc:1261] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260925 03:05:16.996726 27230 raft_consensus.cc:482] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:17.054456 27230 raft_consensus.cc:504] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:17.054983 27625 leader_election.cc:288] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:05:16.997149 27630 raft_consensus.cc:482] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1bc1f2e62ab44af995165bd5116a9d43)
I20260925 03:05:17.056527 27589 leader_election.cc:288] T 1414dcfe53f64a78ae413531eb8ec117 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:16.998972 27515 raft_consensus.cc:482] T 1414dcfe53f64a78ae413531eb8ec117 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:17.057209 27515 raft_consensus.cc:504] T 1414dcfe53f64a78ae413531eb8ec117 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:17.057920 27555 raft_consensus.cc:3032] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 LEADER]: Stepping down as leader of term 9
I20260925 03:05:17.058305 27555 raft_consensus.cc:729] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:05:17.059353 27515 leader_election.cc:288] T 1414dcfe53f64a78ae413531eb8ec117 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:05:17.059092 27555 consensus_queue.cc:260] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:17.060118 27555 raft_consensus.cc:3037] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Advancing to term 10
I20260925 03:05:16.999670 27590 raft_consensus.cc:504] T b627ecb4a5384ef392aa695e06599c39 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:17.061694 27268 leader_election.cc:288] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:17.000576 27671 raft_consensus.cc:686] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 11 LEADER]: Becoming Leader. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:05:17.062263 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc2df40db08649c9a75e90000b878559" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:17.063055 27671 consensus_queue.cc:237] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:17.063560 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abd5b8dc1b7e49e3aa9af900d73ff0e7" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:17.063853 24465 raft_consensus.cc:1226] T 3d89396576884f7c8901dba54f783947 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 5. Current term is 6. Ops: []
I20260925 03:05:17.063493 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5f9b503b0ab43999d582015a367441e" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:17.064443 24469 raft_consensus.cc:3037] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Advancing to term 11
W20260925 03:05:17.015594 24436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 73081.6us
I20260925 03:05:17.043144 27384 consensus_queue.cc:237] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:17.066543 27368 raft_consensus.cc:3037] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Advancing to term 10
I20260925 03:05:17.030983 27485 raft_consensus.cc:482] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:17.062193 27590 leader_election.cc:288] T b627ecb4a5384ef392aa695e06599c39 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 8 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:17.018380 24613 raft_consensus.cc:2354] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 10. Current term is 11.
I20260925 03:05:17.068750 27485 raft_consensus.cc:504] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:05:16.953130 27649 log.cc:924] Time spent T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.078s user 0.000s sys 0.001s
I20260925 03:05:17.070562 24465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74443a0a1dc34d88a2f42e3b722db405" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:17.070832 27485 leader_election.cc:288] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:17.068897 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "550118bc1a0641ce9803bdbf43f2069f" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
W20260925 03:05:17.007323 27508 consensus_meta.cc:287] T 293ef69caf514ac98510a68495f54eda P 869b0a8575b54d08ae2817fad5dd42d5: Time spent flushing consensus metadata: real 0.602s user 0.006s sys 0.005s
I20260925 03:05:17.072964 25197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15861cd28e354598a3c6bfc4337c34bb" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:05:17.025730 24467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21c43af5e6844fdaacc8c625fa01fd8d" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:17.073730 25197 raft_consensus.cc:2379] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 in current term 7: Already voted for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in this term.
I20260925 03:05:17.074210 24467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "100c113a9fd84cc8adf2ec7b33ea0093" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:17.074954 24467 raft_consensus.cc:2379] T 100c113a9fd84cc8adf2ec7b33ea0093 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 4: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:05:17.075618 24467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60d45b7d467d4e1ab92367f6d7d52b15" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 11 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:17.072352 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442d4a29b1d743599f4b6b4871e26668" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:17.043555 27627 consensus_queue.cc:1035] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.033s
I20260925 03:05:17.077787 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b030d84c670b48d5bff337ccc0c77626" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 11 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:17.078593 24473 raft_consensus.cc:2396] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for term 11 because replica has last-logged OpId of term: 10 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 4.
I20260925 03:05:17.028005 24588 leader_election.cc:302] T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:05:17.030309 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e725b82985746b39c0cc8f671c95353" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:17.026716 27603 raft_consensus.cc:2781] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260925 03:05:17.031694 24472 raft_consensus.cc:3037] T 1414dcfe53f64a78ae413531eb8ec117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:05:17.021088 27451 raft_consensus.cc:2781] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Leader election won for term 8
I20260925 03:05:17.026966 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b8e0d260a884ecdb20f79de46c36b19" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:17.060344 24619 raft_consensus.cc:595] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Received request to transfer leadership to TS a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:05:17.080415 24468 raft_consensus.cc:2396] T 7b8e0d260a884ecdb20f79de46c36b19 P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for term 9 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260925 03:05:17.081300 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15861cd28e354598a3c6bfc4337c34bb" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:17.081800 24468 raft_consensus.cc:3037] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:05:16.975234 27385 raft_consensus.cc:504] T edcf6e6a422d4d198d4fbb0f7d3d845d P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:17.011647 27479 raft_consensus.cc:504] T 37a157996e944db2906110d87ecb0c54 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:17.086159 27385 leader_election.cc:288] T edcf6e6a422d4d198d4fbb0f7d3d845d P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:05:17.045898 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:17.076607 24467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1414dcfe53f64a78ae413531eb8ec117" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:17.056175 27630 raft_consensus.cc:504] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:17.086958 24542 raft_consensus.cc:2454] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 9.
I20260925 03:05:16.996492 27488 raft_consensus.cc:482] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:05:17.086344 27479 leader_election.cc:288] T 37a157996e944db2906110d87ecb0c54 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:17.076757 24465 raft_consensus.cc:3037] T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Advancing to term 8
I20260925 03:05:17.087575 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5f9b503b0ab43999d582015a367441e" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:17.089234 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5278a60f97da4e10aecd50a8d9a9a43c" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:17.089977 24473 raft_consensus.cc:2379] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 7: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:05:17.087703 24472 raft_consensus.cc:2454] T 1414dcfe53f64a78ae413531eb8ec117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 5.
I20260925 03:05:17.041637 24466 raft_consensus.cc:3037] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Advancing to term 10
I20260925 03:05:17.079386 27603 raft_consensus.cc:482] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Starting leader election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:05:17.020495 27632 raft_consensus.cc:686] T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 LEADER]: Becoming Leader. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:05:16.868657 27667 raft_consensus.cc:504] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:17.078492 27230 leader_election.cc:288] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:05:17.091578 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7aa47c48404465b94a36fd819fe3c7a" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:17.109699 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f68804eb5a244d87bb395d50683c6012" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:17.080364 24619 raft_consensus.cc:603] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260925 03:05:17.088524 27630 leader_election.cc:288] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:05:17.101709 24467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74443a0a1dc34d88a2f42e3b722db405" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:17.091795 24436 leader_election.cc:302] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: 1bc1f2e62ab44af995165bd5116a9d43, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:17.104002 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d89396576884f7c8901dba54f783947" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:17.144201 24470 raft_consensus.cc:2379] T 3d89396576884f7c8901dba54f783947 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 6: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:05:17.072466 27262 raft_consensus.cc:504] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:17.098431 24468 raft_consensus.cc:2454] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 7.
I20260925 03:05:17.145627 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:17.146260 24468 raft_consensus.cc:2362] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 9. Re-sending same reply.
I20260925 03:05:17.146456 24467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be4e0aa984f4f2b81a58e382e434ad5" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:17.063628 27267 leader_election.cc:288] T 3d89396576884f7c8901dba54f783947 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:17.104449 27603 raft_consensus.cc:3037] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Advancing to term 9
I20260925 03:05:17.060159 27538 leader_election.cc:288] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
W20260925 03:05:17.154881 24436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 63161.5us
I20260925 03:05:17.073360 27508 raft_consensus.cc:2726] T 293ef69caf514ac98510a68495f54eda P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260925 03:05:17.141330 27479 raft_consensus.cc:482] T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:17.136622 27632 consensus_queue.cc:237] T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:17.151751 24469 raft_consensus.cc:2454] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 11.
I20260925 03:05:17.087646 27636 raft_consensus.cc:482] T 0b5df303cd6a4d59aca5c18bbbff24ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
W20260925 03:05:17.155581 27254 consensus_meta.cc:287] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43: Time spent flushing consensus metadata: real 0.930s user 0.007s sys 0.005s
I20260925 03:05:17.158532 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25102ba08e4e49388f6176badcd10a25" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:17.158833 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c85f5d1d3f004cf5817bb594fd8e6469" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:17.182164 24468 raft_consensus.cc:2379] T c85f5d1d3f004cf5817bb594fd8e6469 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 5: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:05:17.160898 24464 raft_consensus.cc:2396] T 293ef69caf514ac98510a68495f54eda P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for term 8 because replica has last-logged OpId of term: 7 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 8.
I20260925 03:05:17.158573 25197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e92f377a79354a6ebb4eec67303c13e3" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:05:17.147111 24467 raft_consensus.cc:2362] T 0be4e0aa984f4f2b81a58e382e434ad5 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 8. Re-sending same reply.
I20260925 03:05:17.158193 27267 raft_consensus.cc:2726] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260925 03:05:17.177207 25199 raft_consensus.cc:1226] T 29637f5a4a504ce6ba29033ccd0cd716 P 1bc1f2e62ab44af995165bd5116a9d43 [term 10 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 9. Current term is 10. Ops: []
I20260925 03:05:17.157106 27634 raft_consensus.cc:482] T 43366a0a2f9d4344bf4e37ffd290ef8a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:17.178614 25163 leader_election.cc:302] T 7e49248643df4131a1c42e74244e05c2 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:05:17.178994 27479 raft_consensus.cc:504] T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:17.183287 25197 raft_consensus.cc:2379] T e92f377a79354a6ebb4eec67303c13e3 P 1bc1f2e62ab44af995165bd5116a9d43 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 8: Already voted for candidate eeab93abfdb141a895002ec9f312296c in this term.
I20260925 03:05:17.170310 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e725b82985746b39c0cc8f671c95353" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:17.182780 24465 raft_consensus.cc:2454] T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 8.
I20260925 03:05:17.154014 27451 raft_consensus.cc:686] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 LEADER]: Becoming Leader. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:05:17.215801 24437 leader_election.cc:302] T 25102ba08e4e49388f6176badcd10a25 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:17.217669 24437 leader_election.cc:302] T 9e8af2641e124f9589c5ccfdac785f63 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:17.189821 24469 raft_consensus.cc:2454] T 1e725b82985746b39c0cc8f671c95353 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 8.
I20260925 03:05:17.190909 25198 raft_consensus.cc:1261] T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43 [term 7 FOLLOWER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260925 03:05:17.190539 27254 raft_consensus.cc:504] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:17.180649 27268 raft_consensus.cc:482] T ac9357d144aa49789faaf31fe5e6bf89 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:17.201877 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc2df40db08649c9a75e90000b878559" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:17.178673 27262 leader_election.cc:288] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 10 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:05:17.103508 27488 raft_consensus.cc:504] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:17.196527 24472 raft_consensus.cc:1261] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260925 03:05:17.209867 24467 raft_consensus.cc:1226] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 6. Current term is 7. Ops: []
I20260925 03:05:17.240087 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1414dcfe53f64a78ae413531eb8ec117" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:17.210376 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367277daf8d04778b3fa4ed94d71580d" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:05:17.211503 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74443a0a1dc34d88a2f42e3b722db405" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
W20260925 03:05:17.180637 27650 log.cc:924] Time spent T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.651s user 0.000s sys 0.003s
I20260925 03:05:17.219228 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b627ecb4a5384ef392aa695e06599c39" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:17.222751 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15861cd28e354598a3c6bfc4337c34bb" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:17.224738 24468 raft_consensus.cc:1226] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [term 10 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 9. Current term is 10. Ops: []
I20260925 03:05:17.243577 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37a157996e944db2906110d87ecb0c54" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:17.245795 24468 raft_consensus.cc:1261] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [term 11 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 10 index: 5. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20260925 03:05:17.218463 27434 raft_consensus.cc:2781] T 7e49248643df4131a1c42e74244e05c2 P 1bc1f2e62ab44af995165bd5116a9d43 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260925 03:05:17.227038 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b627ecb4a5384ef392aa695e06599c39" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:05:17.228145 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d89396576884f7c8901dba54f783947" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:05:17.203727 27629 leader_election.cc:288] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:17.231668 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5f9b503b0ab43999d582015a367441e" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:17.224128 27513 raft_consensus.cc:2726] T 25102ba08e4e49388f6176badcd10a25 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260925 03:05:17.255530 24468 raft_consensus.cc:1261] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260925 03:05:17.221441 24437 leader_election.cc:302] T f22ddd7d1c444dc58f45a1fb0b3e02bc P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:17.223412 27485 raft_consensus.cc:2726] T 9e8af2641e124f9589c5ccfdac785f63 P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
W20260925 03:05:17.262854 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 238862us
I20260925 03:05:17.256569 24514 leader_election.cc:302] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:05:17.265302 27640 raft_consensus.cc:3032] T 11bb46bce3564841b7674e6a693bd2e6 P eeab93abfdb141a895002ec9f312296c [term 5 LEADER]: Stepping down as leader of term 5
I20260925 03:05:17.265702 27640 raft_consensus.cc:729] T 11bb46bce3564841b7674e6a693bd2e6 P eeab93abfdb141a895002ec9f312296c [term 5 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:05:17.266525 27640 consensus_queue.cc:260] T 11bb46bce3564841b7674e6a693bd2e6 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:17.266870 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d89396576884f7c8901dba54f783947" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:05:17.267684 27640 raft_consensus.cc:3037] T 11bb46bce3564841b7674e6a693bd2e6 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:05:17.268314 24620 raft_consensus.cc:1226] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [term 10 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 9. Current term is 10. Ops: []
I20260925 03:05:17.269117 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74443a0a1dc34d88a2f42e3b722db405" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:17.269393 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1414dcfe53f64a78ae413531eb8ec117" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:17.269826 24622 raft_consensus.cc:2379] T 74443a0a1dc34d88a2f42e3b722db405 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 in current term 8: Already voted for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in this term.
I20260925 03:05:17.270076 24613 raft_consensus.cc:2454] T 1414dcfe53f64a78ae413531eb8ec117 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 5.
I20260925 03:05:17.270184 24618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b627ecb4a5384ef392aa695e06599c39" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:05:17.270884 24618 raft_consensus.cc:2379] T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 in current term 8: Already voted for candidate eeab93abfdb141a895002ec9f312296c in this term.
I20260925 03:05:17.254940 24547 raft_consensus.cc:2379] T 1414dcfe53f64a78ae413531eb8ec117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 in current term 5: Already voted for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in this term.
I20260925 03:05:17.256501 24469 raft_consensus.cc:2454] T b627ecb4a5384ef392aa695e06599c39 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 8.
I20260925 03:05:17.273083 24514 leader_election.cc:302] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:17.258654 24437 leader_election.cc:302] T edcf6e6a422d4d198d4fbb0f7d3d845d P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
W20260925 03:05:17.271595 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 243686us
I20260925 03:05:17.260578 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9c33fd100ff44c1a3247319a324a044" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:17.261833 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b5df303cd6a4d59aca5c18bbbff24ed" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:17.223865 27627 raft_consensus.cc:482] T 7993f9c3e2a44f408bbf5d1e395e2306 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:17.263756 24436 leader_election.cc:398] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 10 pre-election: Vote denied by peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105) with higher term. Message: Invalid argument: T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 10. Current term is 11.
I20260925 03:05:17.275519 24436 leader_election.cc:401] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 10 pre-election: Cancelling election due to peer responding with higher term
I20260925 03:05:17.187881 27636 raft_consensus.cc:504] T 0b5df303cd6a4d59aca5c18bbbff24ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:17.276021 24437 leader_election.cc:302] T c88cbd39e01f44628a094af516be2833 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:17.277556 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ac285175464fbda67211d9574db415" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:17.278817 24514 leader_election.cc:302] T 550118bc1a0641ce9803bdbf43f2069f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:17.278761 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b8e0d260a884ecdb20f79de46c36b19" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:17.274559 24465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f1fac028a144058b92331c0d9b44ebf" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:17.279400 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1414dcfe53f64a78ae413531eb8ec117" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:17.281016 24514 leader_election.cc:302] T 21c43af5e6844fdaacc8c625fa01fd8d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:17.280761 24468 raft_consensus.cc:2454] T 7b8e0d260a884ecdb20f79de46c36b19 P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 9.
I20260925 03:05:17.253144 27434 raft_consensus.cc:482] T 7e49248643df4131a1c42e74244e05c2 P 1bc1f2e62ab44af995165bd5116a9d43 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:05:17.208006 27634 raft_consensus.cc:504] T 43366a0a2f9d4344bf4e37ffd290ef8a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:17.283818 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceab6b0fccd54e909a64f66fe81a9a1f" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:17.287410 24468 raft_consensus.cc:3037] T ceab6b0fccd54e909a64f66fe81a9a1f P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Advancing to term 10
I20260925 03:05:17.261484 27589 raft_consensus.cc:2726] T f22ddd7d1c444dc58f45a1fb0b3e02bc P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260925 03:05:17.284125 27675 raft_consensus.cc:2726] T 21c43af5e6844fdaacc8c625fa01fd8d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260925 03:05:17.285354 24586 leader_election.cc:302] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:05:17.308792 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcf6e6a422d4d198d4fbb0f7d3d845d" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
W20260925 03:05:17.309017 24436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 45408.4us
I20260925 03:05:17.310283 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37a157996e944db2906110d87ecb0c54" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:17.285264 24514 leader_election.cc:302] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:17.278771 27543 raft_consensus.cc:2726] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260925 03:05:17.281456 24465 raft_consensus.cc:2396] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 for term 9 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 5.
I20260925 03:05:17.281463 24469 raft_consensus.cc:2454] T 1414dcfe53f64a78ae413531eb8ec117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 5.
I20260925 03:05:17.288386 24542 raft_consensus.cc:2454] T 0b5df303cd6a4d59aca5c18bbbff24ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 8.
I20260925 03:05:17.289119 27451 consensus_queue.cc:237] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:17.289955 27637 raft_consensus.cc:2726] T c88cbd39e01f44628a094af516be2833 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260925 03:05:17.344642 24615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b5df303cd6a4d59aca5c18bbbff24ed" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:17.282363 27460 raft_consensus.cc:2726] T 550118bc1a0641ce9803bdbf43f2069f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260925 03:05:17.291842 27401 raft_consensus.cc:979] T 3d89396576884f7c8901dba54f783947 P eeab93abfdb141a895002ec9f312296c: : Instructing follower a04a37fc4aa7460cbfdbeeb4e451e1f5 to start an election
I20260925 03:05:17.282133 27673 raft_consensus.cc:2726] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260925 03:05:17.284610 27385 consensus_queue.cc:1035] T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.008s
I20260925 03:05:17.270838 27629 consensus_queue.cc:1046] T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 5.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.022s
I20260925 03:05:17.280709 27513 raft_consensus.cc:2781] T edcf6e6a422d4d198d4fbb0f7d3d845d P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader election won for term 6
I20260925 03:05:17.288843 27384 raft_consensus.cc:482] T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:17.279985 27627 raft_consensus.cc:504] T 7993f9c3e2a44f408bbf5d1e395e2306 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:17.293846 27667 leader_election.cc:288] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:17.295913 27636 leader_election.cc:288] T 0b5df303cd6a4d59aca5c18bbbff24ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:17.261119 27631 consensus_queue.cc:1035] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.019s
I20260925 03:05:17.239741 27268 raft_consensus.cc:504] T ac9357d144aa49789faaf31fe5e6bf89 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:17.284162 24543 raft_consensus.cc:2454] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 8.
I20260925 03:05:17.306226 24619 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "abd5b8dc1b7e49e3aa9af900d73ff0e7"
dest_uuid: "eeab93abfdb141a895002ec9f312296c"
mode: GRACEFUL
new_leader_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
from {username='slave'} at 127.0.0.1:44570
I20260925 03:05:17.277269 27515 raft_consensus.cc:2781] T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260925 03:05:17.288560 24437 leader_election.cc:302] T 11bb46bce3564841b7674e6a693bd2e6 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:17.312026 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "420163a6cdce412ebc21c25fe3835638" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:17.336633 24540 raft_consensus.cc:1261] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260925 03:05:17.336998 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ac285175464fbda67211d9574db415" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
W20260925 03:05:17.330757 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 45567.7us
I20260925 03:05:17.341300 24618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b627ecb4a5384ef392aa695e06599c39" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:17.341686 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be4e0aa984f4f2b81a58e382e434ad5" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:17.342192 27414 leader_election.cc:288] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:17.380471 24618 raft_consensus.cc:2454] T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 8.
I20260925 03:05:17.382998 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ac466793d094325ace224708b2158ed" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:17.342132 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b8e0d260a884ecdb20f79de46c36b19" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:17.344995 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ac285175464fbda67211d9574db415" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:17.290628 25198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ac466793d094325ace224708b2158ed" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:05:17.345952 24468 raft_consensus.cc:2454] T ceab6b0fccd54e909a64f66fe81a9a1f P 869b0a8575b54d08ae2817fad5dd42d5 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 10.
I20260925 03:05:17.346417 24620 raft_consensus.cc:1261] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260925 03:05:17.384002 24539 raft_consensus.cc:2454] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 9.
I20260925 03:05:17.286659 27434 raft_consensus.cc:3037] T 7e49248643df4131a1c42e74244e05c2 P 1bc1f2e62ab44af995165bd5116a9d43 [term 9 FOLLOWER]: Advancing to term 10
I20260925 03:05:17.370404 27241 raft_consensus.cc:1067] T 3d89396576884f7c8901dba54f783947 P eeab93abfdb141a895002ec9f312296c [term 5 LEADER]: Signalling peer a04a37fc4aa7460cbfdbeeb4e451e1f5 to start an election
I20260925 03:05:17.381577 27675 raft_consensus.cc:2726] T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260925 03:05:17.345579 24436 leader_election.cc:302] T 1414dcfe53f64a78ae413531eb8ec117 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:05:17.351393 27488 leader_election.cc:288] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:05:17.355618 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5332a5d329493991dff994e70ea92c" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:17.314668 27328 raft_consensus.cc:2726] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260925 03:05:17.363188 24614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b5df303cd6a4d59aca5c18bbbff24ed" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:17.364158 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcf6e6a422d4d198d4fbb0f7d3d845d" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:17.369843 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1414dcfe53f64a78ae413531eb8ec117" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:17.314751 27664 raft_consensus.cc:2726] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [term 11 FOLLOWER]: Leader pre-election lost for term 10. Reason: Vote denied by peer eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105) with higher term. Message: Invalid argument: T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 10. Current term is 11.
I20260925 03:05:17.354848 27638 raft_consensus.cc:482] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:17.372649 25199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:05:17.375036 24619 raft_consensus.cc:595] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P eeab93abfdb141a895002ec9f312296c [term 8 LEADER]: Received request to transfer leadership to TS a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:05:17.375205 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceab6b0fccd54e909a64f66fe81a9a1f" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:05:17.398365 24543 raft_consensus.cc:3037] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Advancing to term 10
I20260925 03:05:17.399608 27634 leader_election.cc:288] T 43366a0a2f9d4344bf4e37ffd290ef8a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:17.399296 27638 raft_consensus.cc:504] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:17.390319 27667 raft_consensus.cc:2741] T 1414dcfe53f64a78ae413531eb8ec117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260925 03:05:17.380306 24548 raft_consensus.cc:2454] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 9.
I20260925 03:05:17.380654 24547 raft_consensus.cc:2454] T 0be4e0aa984f4f2b81a58e382e434ad5 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 8.
I20260925 03:05:17.380611 24514 leader_election.cc:302] T 442d4a29b1d743599f4b6b4871e26668 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
W20260925 03:05:17.388276 24436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 50591.7us
I20260925 03:05:17.366932 27673 raft_consensus.cc:3032] T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 LEADER]: Stepping down as leader of term 8
I20260925 03:05:17.357752 27538 consensus_queue.cc:1046] T 3d89396576884f7c8901dba54f783947 P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.020s
I20260925 03:05:17.368327 27513 raft_consensus.cc:686] T edcf6e6a422d4d198d4fbb0f7d3d845d P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 LEADER]: Becoming Leader. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:05:17.355029 27622 consensus_queue.cc:1035] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.024s
I20260925 03:05:17.368835 27254 leader_election.cc:288] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 9 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:05:17.396843 24544 raft_consensus.cc:2454] T edcf6e6a422d4d198d4fbb0f7d3d845d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 6.
I20260925 03:05:17.396971 24542 raft_consensus.cc:2454] T 1414dcfe53f64a78ae413531eb8ec117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 5.
I20260925 03:05:17.400866 25198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e068a0f80941dc9138cee063f53bab" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:05:17.405875 24543 raft_consensus.cc:2454] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 10.
I20260925 03:05:17.403378 24514 leader_election.cc:302] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
W20260925 03:05:17.380030 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 91527.3us
I20260925 03:05:17.407624 24437 leader_election.cc:302] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:17.407940 24514 leader_election.cc:302] T b030d84c670b48d5bff337ccc0c77626 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:17.382350 27515 raft_consensus.cc:482] T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:05:17.388196 27628 raft_consensus.cc:2726] T 11bb46bce3564841b7674e6a693bd2e6 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260925 03:05:17.409017 25199 raft_consensus.cc:2379] T 29637f5a4a504ce6ba29033ccd0cd716 P 1bc1f2e62ab44af995165bd5116a9d43 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c in current term 10: Already voted for candidate 1bc1f2e62ab44af995165bd5116a9d43 in this term.
I20260925 03:05:17.387487 27268 leader_election.cc:288] T ac9357d144aa49789faaf31fe5e6bf89 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:17.368104 27384 raft_consensus.cc:504] T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:17.411295 24437 leader_election.cc:302] T 100c113a9fd84cc8adf2ec7b33ea0093 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:17.413900 24514 leader_election.cc:302] T 5278a60f97da4e10aecd50a8d9a9a43c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:17.420535 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "293ef69caf514ac98510a68495f54eda" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:17.425738 24468 raft_consensus.cc:3037] T 293ef69caf514ac98510a68495f54eda P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Advancing to term 9
I20260925 03:05:17.426769 24544 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3d89396576884f7c8901dba54f783947"
dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
from {username='slave'} at 127.0.0.1:36786
I20260925 03:05:17.427379 24544 raft_consensus.cc:482] T 3d89396576884f7c8901dba54f783947 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 03:05:17.427698 24544 raft_consensus.cc:3037] T 3d89396576884f7c8901dba54f783947 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:05:17.431825 27384 leader_election.cc:288] T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:05:17.414528 27515 raft_consensus.cc:3037] T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Advancing to term 9
I20260925 03:05:17.433014 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:17.433686 24542 raft_consensus.cc:2454] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 10.
I20260925 03:05:17.426188 24514 leader_election.cc:302] T 1414dcfe53f64a78ae413531eb8ec117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:17.436197 24466 raft_consensus.cc:2454] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 10.
I20260925 03:05:17.436930 24514 leader_election.cc:302] T a7aa47c48404465b94a36fd819fe3c7a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:17.435855 24437 leader_election.cc:302] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:17.406172 27634 raft_consensus.cc:2726] T 442d4a29b1d743599f4b6b4871e26668 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260925 03:05:17.405177 27673 raft_consensus.cc:729] T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:05:17.439530 27673 consensus_queue.cc:260] T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:17.440834 27673 raft_consensus.cc:3037] T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Advancing to term 9
I20260925 03:05:17.444720 27636 raft_consensus.cc:3037] T 5278a60f97da4e10aecd50a8d9a9a43c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:05:17.438120 24514 leader_election.cc:302] T 3d89396576884f7c8901dba54f783947 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:17.438503 24437 leader_election.cc:302] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:17.446758 24437 leader_election.cc:302] T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:05:17.445111 27628 raft_consensus.cc:2726] T 100c113a9fd84cc8adf2ec7b33ea0093 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260925 03:05:17.437196 24464 raft_consensus.cc:1261] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [term 10 FOLLOWER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 10 index: 5. (index mismatch)
I20260925 03:05:17.416970 27671 raft_consensus.cc:2726] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260925 03:05:17.455186 24614 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c85f5d1d3f004cf5817bb594fd8e6469"
dest_uuid: "eeab93abfdb141a895002ec9f312296c"
mode: GRACEFUL
new_leader_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
from {username='slave'} at 127.0.0.1:44570
I20260925 03:05:17.455878 24614 raft_consensus.cc:595] T c85f5d1d3f004cf5817bb594fd8e6469 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Received request to transfer leadership to TS a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:05:17.456231 24614 raft_consensus.cc:603] T c85f5d1d3f004cf5817bb594fd8e6469 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260925 03:05:17.448421 24586 leader_election.cc:302] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:05:17.447520 24514 leader_election.cc:302] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters: 1bc1f2e62ab44af995165bd5116a9d43
I20260925 03:05:17.401852 27479 leader_election.cc:288] T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:05:17.412135 27667 raft_consensus.cc:482] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:17.437599 24544 raft_consensus.cc:504] T 3d89396576884f7c8901dba54f783947 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:17.464797 24514 leader_election.cc:302] T 25102ba08e4e49388f6176badcd10a25 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:17.461440 24468 raft_consensus.cc:2454] T 293ef69caf514ac98510a68495f54eda P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 9.
I20260925 03:05:17.447477 24543 raft_consensus.cc:1261] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260925 03:05:17.449810 24437 leader_election.cc:302] T 8e5819c15eac4544a57e5ee6c9d4f117 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:17.466463 25165 leader_election.cc:302] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5; no voters:
I20260925 03:05:17.417448 27664 raft_consensus.cc:2726] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260925 03:05:17.445708 27637 raft_consensus.cc:2726] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260925 03:05:17.417846 27631 raft_consensus.cc:2726] T b030d84c670b48d5bff337ccc0c77626 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260925 03:05:17.463337 24586 leader_election.cc:302] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eeab93abfdb141a895002ec9f312296c; no voters: 1bc1f2e62ab44af995165bd5116a9d43, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:05:17.466094 24544 leader_election.cc:288] T 3d89396576884f7c8901dba54f783947 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 6 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
W20260925 03:05:17.468412 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 99327.3us
I20260925 03:05:17.472594 24437 leader_election.cc:302] T c85f5d1d3f004cf5817bb594fd8e6469 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:05:17.475126 24437 leader_election.cc:302] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: 1bc1f2e62ab44af995165bd5116a9d43, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:05:17.448331 27629 raft_consensus.cc:2726] T a7aa47c48404465b94a36fd819fe3c7a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260925 03:05:17.447940 27267 raft_consensus.cc:2726] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260925 03:05:17.414752 27485 consensus_queue.cc:1035] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.046s
I20260925 03:05:17.449077 27589 raft_consensus.cc:2781] T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260925 03:05:17.471092 24514 leader_election.cc:302] T 0be4e0aa984f4f2b81a58e382e434ad5 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:05:17.452085 27414 raft_consensus.cc:2726] T 1414dcfe53f64a78ae413531eb8ec117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260925 03:05:17.477974 24437 leader_election.cc:302] T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:17.481606 24547 raft_consensus.cc:1261] T b030d84c670b48d5bff337ccc0c77626 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 11 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20260925 03:05:17.463120 27040 raft_consensus.cc:2781] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260925 03:05:17.466866 27628 raft_consensus.cc:2726] T 8e5819c15eac4544a57e5ee6c9d4f117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260925 03:05:17.467993 27268 raft_consensus.cc:2726] T 3d89396576884f7c8901dba54f783947 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260925 03:05:17.489883 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "293ef69caf514ac98510a68495f54eda" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:05:17.469707 27634 raft_consensus.cc:2781] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Leader election won for term 9
I20260925 03:05:17.493096 24545 raft_consensus.cc:3037] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Advancing to term 9
I20260925 03:05:17.477165 27609 raft_consensus.cc:2781] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43 [term 9 FOLLOWER]: Leader election won for term 9
I20260925 03:05:17.477100 27479 raft_consensus.cc:2726] T 25102ba08e4e49388f6176badcd10a25 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260925 03:05:17.487389 24437 leader_election.cc:302] T 3d89396576884f7c8901dba54f783947 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:17.486174 24586 leader_election.cc:302] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:05:17.488838 27622 raft_consensus.cc:2726] T 8ac466793d094325ace224708b2158ed P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260925 03:05:17.501748 27684 log.cc:924] Time spent T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.085s user 0.000s sys 0.000s
I20260925 03:05:17.481827 27427 raft_consensus.cc:2726] T 15861cd28e354598a3c6bfc4337c34bb P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260925 03:05:17.510010 24437 leader_election.cc:302] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:05:17.483604 27670 raft_consensus.cc:2781] T c85f5d1d3f004cf5817bb594fd8e6469 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader election won for term 5
I20260925 03:05:17.461942 27640 consensus_queue.cc:1035] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.026s
I20260925 03:05:17.513110 24586 leader_election.cc:302] T edcf6e6a422d4d198d4fbb0f7d3d845d P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:05:17.490048 27625 consensus_queue.cc:1035] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.028s
I20260925 03:05:17.469745 27667 raft_consensus.cc:504] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:17.491677 27267 raft_consensus.cc:2726] T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260925 03:05:17.484520 27589 raft_consensus.cc:482] T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:17.518169 24437 leader_election.cc:302] T b627ecb4a5384ef392aa695e06599c39 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:05:17.489322 27040 raft_consensus.cc:482] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:05:17.510951 24614 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f68804eb5a244d87bb395d50683c6012"
dest_uuid: "eeab93abfdb141a895002ec9f312296c"
mode: GRACEFUL
new_leader_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
from {username='slave'} at 127.0.0.1:44570
I20260925 03:05:17.509953 27634 raft_consensus.cc:686] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 LEADER]: Becoming Leader. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:05:17.491191 27629 raft_consensus.cc:2781] T 0be4e0aa984f4f2b81a58e382e434ad5 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Leader election won for term 8
I20260925 03:05:17.516031 27609 raft_consensus.cc:686] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43 [term 9 LEADER]: Becoming Leader. State: Replica: 1bc1f2e62ab44af995165bd5116a9d43, State: Running, Role: LEADER
I20260925 03:05:17.525583 24586 leader_election.cc:302] T 1414dcfe53f64a78ae413531eb8ec117 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:05:17.543192 24465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b5df303cd6a4d59aca5c18bbbff24ed" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:17.544523 24465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d89396576884f7c8901dba54f783947" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: true dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:17.540122 27427 raft_consensus.cc:2781] T b627ecb4a5384ef392aa695e06599c39 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Leader election won for term 8
I20260925 03:05:17.545647 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ac285175464fbda67211d9574db415" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:17.546941 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7993f9c3e2a44f408bbf5d1e395e2306" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:17.548135 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9357d144aa49789faaf31fe5e6bf89" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:17.515040 27384 raft_consensus.cc:2781] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260925 03:05:17.515673 27590 raft_consensus.cc:2726] T 3d89396576884f7c8901dba54f783947 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260925 03:05:17.528416 24614 raft_consensus.cc:595] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Received request to transfer leadership to TS a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:05:17.524420 24545 raft_consensus.cc:2454] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 9.
I20260925 03:05:17.552000 24614 raft_consensus.cc:603] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260925 03:05:17.533825 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "420163a6cdce412ebc21c25fe3835638" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:17.535385 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "100c113a9fd84cc8adf2ec7b33ea0093" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:17.553176 24469 raft_consensus.cc:2379] T 100c113a9fd84cc8adf2ec7b33ea0093 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 4: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:05:17.522559 27627 leader_election.cc:288] T 7993f9c3e2a44f408bbf5d1e395e2306 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:17.537367 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06f4a7eaf6534e4986c9fd35911038e4" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:17.533730 24514 leader_election.cc:302] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:05:17.557902 24514 leader_election.cc:302] T cc2df40db08649c9a75e90000b878559 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:17.529549 27622 raft_consensus.cc:2781] T edcf6e6a422d4d198d4fbb0f7d3d845d P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260925 03:05:17.535935 24467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7691c6b475428c82d6181d693d26f3" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:17.562248 24467 raft_consensus.cc:2379] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 9: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:05:17.521538 27628 raft_consensus.cc:2781] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260925 03:05:17.529520 27589 raft_consensus.cc:3037] T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:05:17.523409 27670 raft_consensus.cc:686] T c85f5d1d3f004cf5817bb594fd8e6469 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 LEADER]: Becoming Leader. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:05:17.549059 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5332a5d329493991dff994e70ea92c" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:17.565840 24514 leader_election.cc:302] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:05:17.566078 24466 raft_consensus.cc:2454] T 4e5332a5d329493991dff994e70ea92c P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 9.
I20260925 03:05:17.566237 23300 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260925 03:05:17.567225 24514 leader_election.cc:302] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:17.490316 27513 consensus_queue.cc:237] T edcf6e6a422d4d198d4fbb0f7d3d845d P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:17.559072 24465 raft_consensus.cc:2379] T 3d89396576884f7c8901dba54f783947 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 6: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:05:17.559598 24586 leader_election.cc:302] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:05:17.539283 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9c33fd100ff44c1a3247319a324a044" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:17.568216 23300 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260925 03:05:17.531885 27040 raft_consensus.cc:3037] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Advancing to term 10
I20260925 03:05:17.561903 27230 raft_consensus.cc:3032] T 3d89396576884f7c8901dba54f783947 P eeab93abfdb141a895002ec9f312296c [term 5 LEADER]: Stepping down as leader of term 5
I20260925 03:05:17.568531 24468 raft_consensus.cc:2362] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 10. Re-sending same reply.
I20260925 03:05:17.570655 23300 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260925 03:05:17.571414 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43366a0a2f9d4344bf4e37ffd290ef8a" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:17.553639 27384 raft_consensus.cc:482] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:05:17.553400 27427 raft_consensus.cc:686] T b627ecb4a5384ef392aa695e06599c39 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 LEADER]: Becoming Leader. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:05:17.553874 27629 raft_consensus.cc:686] T 0be4e0aa984f4f2b81a58e382e434ad5 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 LEADER]: Becoming Leader. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:05:17.572937 27230 raft_consensus.cc:729] T 3d89396576884f7c8901dba54f783947 P eeab93abfdb141a895002ec9f312296c [term 5 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:05:17.563863 27631 raft_consensus.cc:2726] T cc2df40db08649c9a75e90000b878559 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260925 03:05:17.564661 27627 raft_consensus.cc:2781] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260925 03:05:17.561896 27622 raft_consensus.cc:482] T edcf6e6a422d4d198d4fbb0f7d3d845d P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260925 03:05:17.587357 24367 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:57656 (request call id 85) took 12169 ms (client timeout 15000 ms). Trace:
I20260925 03:05:17.591621 27666 raft_consensus.cc:482] T 29637f5a4a504ce6ba29033ccd0cd716 P 1bc1f2e62ab44af995165bd5116a9d43 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:17.569057 27628 raft_consensus.cc:482] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:17.563187 27686 raft_consensus.cc:2781] T 1414dcfe53f64a78ae413531eb8ec117 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260925 03:05:17.599749 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9c33fd100ff44c1a3247319a324a044" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 11 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:17.575562 27460 raft_consensus.cc:2781] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260925 03:05:17.581918 27384 raft_consensus.cc:3037] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Advancing to term 10
I20260925 03:05:17.582652 27479 raft_consensus.cc:2726] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260925 03:05:17.543556 27609 consensus_queue.cc:237] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 7.9, Last appended by leader: 9, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:17.586954 27328 raft_consensus.cc:2781] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [term 10 FOLLOWER]: Leader election won for term 10
I20260925 03:05:17.624984 27328 raft_consensus.cc:686] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [term 10 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:05:17.625893 27328 consensus_queue.cc:237] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:17.630908 27427 consensus_queue.cc:237] T b627ecb4a5384ef392aa695e06599c39 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:17.607107 27630 raft_consensus.cc:482] T 100c113a9fd84cc8adf2ec7b33ea0093 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:17.607581 27622 raft_consensus.cc:3037] T edcf6e6a422d4d198d4fbb0f7d3d845d P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:05:17.612295 27460 raft_consensus.cc:482] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:05:17.607602 27262 raft_consensus.cc:482] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:17.609658 27640 consensus_queue.cc:1035] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.013s
I20260925 03:05:17.607671 27488 raft_consensus.cc:482] T 5278a60f97da4e10aecd50a8d9a9a43c P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:17.612414 27634 consensus_queue.cc:237] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 9, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:17.611289 27628 raft_consensus.cc:3037] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Advancing to term 9
I20260925 03:05:17.600258 27627 raft_consensus.cc:482] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:05:17.640642 24514 leader_election.cc:302] T 100c113a9fd84cc8adf2ec7b33ea0093 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:17.491849 27635 consensus_queue.cc:1046] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.178s
I20260925 03:05:17.501569 27687 consensus_queue.cc:1035] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.062s
I20260925 03:05:17.645262 27230 consensus_queue.cc:260] T 3d89396576884f7c8901dba54f783947 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:17.572377 27671 consensus_queue.cc:1035] T 15861cd28e354598a3c6bfc4337c34bb P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.035s
I20260925 03:05:17.572542 27414 consensus_queue.cc:1035] T 420163a6cdce412ebc21c25fe3835638 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.035s
I20260925 03:05:17.492313 27669 consensus_queue.cc:1046] T 442d4a29b1d743599f4b6b4871e26668 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.078s
I20260925 03:05:17.669850 27669 raft_consensus.cc:3032] T 442d4a29b1d743599f4b6b4871e26668 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 LEADER]: Stepping down as leader of term 8
W20260925 03:05:17.670185 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 177493us
I20260925 03:05:17.670228 27669 raft_consensus.cc:729] T 442d4a29b1d743599f4b6b4871e26668 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:05:17.672490 27669 consensus_queue.cc:260] T 442d4a29b1d743599f4b6b4871e26668 P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:17.678017 27401 consensus_queue.cc:1046] T 367277daf8d04778b3fa4ed94d71580d P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260925 03:05:17.679702 27241 raft_consensus.cc:3032] T 367277daf8d04778b3fa4ed94d71580d P eeab93abfdb141a895002ec9f312296c [term 6 LEADER]: Stepping down as leader of term 6
I20260925 03:05:17.680065 27241 raft_consensus.cc:729] T 367277daf8d04778b3fa4ed94d71580d P eeab93abfdb141a895002ec9f312296c [term 6 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
W20260925 03:05:17.605050 24367 rpcz_store.cc:269] 0925 03:05:05.418151 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:05:05.418397 (+ 246us) service_pool.cc:227] Handling call
0925 03:05:17.587290 (+12168893us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0925 03:05:15.360821 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0925 03:05:15.411926 (+ 51105us) write_op.cc:432] Acquiring schema lock in shared mode
0925 03:05:15.411978 (+ 52us) write_op.cc:435] Acquired schema lock
0925 03:05:15.411993 (+ 15us) tablet.cc:660] Decoding operations
0925 03:05:15.412947 (+ 954us) write_op.cc:620] Acquiring the partition lock for write op
0925 03:05:15.413036 (+ 89us) write_op.cc:641] Partition lock acquired for write op
0925 03:05:15.413059 (+ 23us) tablet.cc:683] Acquiring locks for 9 operations
0925 03:05:15.413789 (+ 730us) tablet.cc:699] Row locks acquired
0925 03:05:15.413827 (+ 38us) write_op.cc:260] PREPARE: finished
0925 03:05:15.413987 (+ 160us) write_op.cc:270] Start()
0925 03:05:15.414125 (+ 138us) write_op.cc:276] Timestamp: P: 1790305515413922 usec, L: 0
0925 03:05:15.414161 (+ 36us) op_driver.cc:331] REPLICATION: starting
0925 03:05:15.414837 (+ 676us) log.cc:842] Serialized 3631 byte log entry
0925 03:05:16.108991 (+694154us) spinlock_profiling.cc:243] Waited 13.9 ms on lock 0x7b44000232cc. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f43733ecf11 00007f43734baa95 00007f43734bb4b2 00007f4376163511 00007f437615ee1a 00007f4376152bee 00007f4376151f36 00007f4376151b58 00007f437615b551 00007f437615b509 00007f437615b499
0925 03:05:16.922187 (+813196us) op_driver.cc:444] REPLICATION: finished
0925 03:05:17.197022 (+274835us) write_op.cc:301] APPLY: starting
0925 03:05:17.197130 (+ 108us) tablet.cc:1365] starting BulkCheckPresence
0925 03:05:17.197403 (+ 273us) tablet.cc:1368] finished BulkCheckPresence
0925 03:05:17.197426 (+ 23us) tablet.cc:1370] starting ApplyRowOperation cycle
0925 03:05:17.198730 (+ 1304us) tablet.cc:1381] finished ApplyRowOperation cycle
0925 03:05:17.199038 (+ 308us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=9
0925 03:05:17.199088 (+ 50us) write_op.cc:312] APPLY: finished
0925 03:05:17.199469 (+ 381us) log.cc:842] Serialized 70 byte log entry
0925 03:05:17.199678 (+ 209us) write_op.cc:489] Releasing partition, row and schema locks
0925 03:05:17.199807 (+ 129us) write_op.cc:454] Released schema lock
0925 03:05:17.199870 (+ 63us) write_op.cc:341] FINISH: Updating metrics
0925 03:05:17.200157 (+ 287us) write_op.cc:489] Releasing partition, row and schema locks
0925 03:05:17.200188 (+ 31us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":27105,"reported_tablets":15,"child_traces":[["op",{"apply.queue_time_us":253195,"apply.run_cpu_time_us":3521,"apply.run_wall_time_us":3522,"num_ops":9,"prepare.queue_time_us":436710,"prepare.run_cpu_time_us":31769,"prepare.run_wall_time_us":320109,"raft.queue_time_us":556218,"raft.run_cpu_time_us":39371,"raft.run_wall_time_us":278159,"replication_time_us":1475063,"spinlock_wait_cycles":31973632,"thread_start_us":763856,"threads_started":4,"wal-append.queue_time_us":264149}]]}
I20260925 03:05:17.611387 27686 raft_consensus.cc:482] T 1414dcfe53f64a78ae413531eb8ec117 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:05:17.606706 27670 consensus_queue.cc:237] T c85f5d1d3f004cf5817bb594fd8e6469 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:17.606901 27603 raft_consensus.cc:504] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:17.692279 27230 raft_consensus.cc:3037] T 3d89396576884f7c8901dba54f783947 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:05:17.657248 27488 raft_consensus.cc:504] T 5278a60f97da4e10aecd50a8d9a9a43c P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:17.693198 27603 leader_election.cc:288] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 9 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:05:17.694439 27488 leader_election.cc:288] T 5278a60f97da4e10aecd50a8d9a9a43c P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
W20260925 03:05:17.696151 27685 log.cc:924] Time spent T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.220s user 0.002s sys 0.000s
W20260925 03:05:17.697180 27555 consensus_meta.cc:287] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5: Time spent flushing consensus metadata: real 0.637s user 0.012s sys 0.002s
I20260925 03:05:17.656246 27262 raft_consensus.cc:504] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:17.699542 27262 leader_election.cc:288] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:05:17.700666 25199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:05:17.651111 27460 raft_consensus.cc:3037] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Advancing to term 9
I20260925 03:05:17.612633 27666 raft_consensus.cc:504] T 29637f5a4a504ce6ba29033ccd0cd716 P 1bc1f2e62ab44af995165bd5116a9d43 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:17.647188 27434 raft_consensus.cc:504] T 7e49248643df4131a1c42e74244e05c2 P 1bc1f2e62ab44af995165bd5116a9d43 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:17.665023 27627 raft_consensus.cc:3037] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Advancing to term 9
I20260925 03:05:17.715704 24614 raft_consensus.cc:1261] T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260925 03:05:17.703806 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5278a60f97da4e10aecd50a8d9a9a43c" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:17.704344 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f22ddd7d1c444dc58f45a1fb0b3e02bc" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:05:17.706317 27667 leader_election.cc:288] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:17.731552 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7691c6b475428c82d6181d693d26f3" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:17.705574 24511 leader_election.cc:302] T 37a157996e944db2906110d87ecb0c54 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:17.709012 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:17.733146 24468 raft_consensus.cc:2379] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c in current term 10: Already voted for candidate 1bc1f2e62ab44af995165bd5116a9d43 in this term.
I20260925 03:05:17.734438 24543 raft_consensus.cc:2454] T f22ddd7d1c444dc58f45a1fb0b3e02bc P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 9.
I20260925 03:05:17.710752 24473 raft_consensus.cc:1261] T 4e5332a5d329493991dff994e70ea92c P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260925 03:05:17.680891 27241 consensus_queue.cc:260] T 367277daf8d04778b3fa4ed94d71580d P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:17.650000 27630 raft_consensus.cc:504] T 100c113a9fd84cc8adf2ec7b33ea0093 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:17.742897 24614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9357d144aa49789faaf31fe5e6bf89" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:17.743106 24473 raft_consensus.cc:1261] T b5f9b503b0ab43999d582015a367441e P 869b0a8575b54d08ae2817fad5dd42d5 [term 11 FOLLOWER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20260925 03:05:17.743261 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b030d84c670b48d5bff337ccc0c77626" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:17.744232 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43366a0a2f9d4344bf4e37ffd290ef8a" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
W20260925 03:05:17.719714 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 79141.6us
I20260925 03:05:17.744168 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d89396576884f7c8901dba54f783947" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: true dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:05:17.699434 27638 leader_election.cc:288] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:05:17.745368 24514 leader_election.cc:302] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:17.745541 25199 raft_consensus.cc:2454] T 29637f5a4a504ce6ba29033ccd0cd716 P 1bc1f2e62ab44af995165bd5116a9d43 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 10.
I20260925 03:05:17.746570 24514 leader_election.cc:302] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:17.704751 24540 raft_consensus.cc:1261] T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260925 03:05:17.747800 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f22ddd7d1c444dc58f45a1fb0b3e02bc" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:17.749743 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e068a0f80941dc9138cee063f53bab" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:17.754559 24473 raft_consensus.cc:2379] T f22ddd7d1c444dc58f45a1fb0b3e02bc P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c in current term 9: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:05:17.705626 27686 raft_consensus.cc:3037] T 1414dcfe53f64a78ae413531eb8ec117 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:05:17.762881 24588 leader_election.cc:302] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:05:17.722880 24437 leader_election.cc:302] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:17.737886 24586 leader_election.cc:302] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:05:17.739574 24511 leader_election.cc:302] T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 1bc1f2e62ab44af995165bd5116a9d43, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:17.732276 24539 raft_consensus.cc:2454] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 9.
I20260925 03:05:17.741051 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9c33fd100ff44c1a3247319a324a044" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 11 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
W20260925 03:05:17.767326 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 44544.4us
I20260925 03:05:17.767800 24472 raft_consensus.cc:2454] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 10.
I20260925 03:05:17.771484 24511 leader_election.cc:302] T 0b5df303cd6a4d59aca5c18bbbff24ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:17.773597 24511 leader_election.cc:302] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:17.779685 24465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5278a60f97da4e10aecd50a8d9a9a43c" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:17.780366 24465 raft_consensus.cc:2454] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 7.
I20260925 03:05:17.782747 24588 leader_election.cc:302] T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:05:17.776397 24618 raft_consensus.cc:1261] T b5f9b503b0ab43999d582015a367441e P eeab93abfdb141a895002ec9f312296c [term 11 FOLLOWER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20260925 03:05:17.745826 27543 raft_consensus.cc:2726] T 37a157996e944db2906110d87ecb0c54 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260925 03:05:17.779579 24616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:17.742067 27414 raft_consensus.cc:2726] T 100c113a9fd84cc8adf2ec7b33ea0093 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260925 03:05:17.780171 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7993f9c3e2a44f408bbf5d1e395e2306" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
W20260925 03:05:17.777566 27685 log.cc:924] Time spent T 60d45b7d467d4e1ab92367f6d7d52b15 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.058s user 0.000s sys 0.000s
I20260925 03:05:17.752315 27667 raft_consensus.cc:482] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:17.787600 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7691c6b475428c82d6181d693d26f3" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:17.804080 24473 raft_consensus.cc:2454] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 9.
I20260925 03:05:17.756769 27632 raft_consensus.cc:2726] T 06f4a7eaf6534e4986c9fd35911038e4 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260925 03:05:17.666673 27629 consensus_queue.cc:237] T 0be4e0aa984f4f2b81a58e382e434ad5 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:17.772089 27638 raft_consensus.cc:2726] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260925 03:05:17.770376 27640 raft_consensus.cc:3037] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Advancing to term 10
I20260925 03:05:17.777181 27488 consensus_queue.cc:1035] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.034s
I20260925 03:05:17.822996 25197 raft_consensus.cc:1261] T e92f377a79354a6ebb4eec67303c13e3 P 1bc1f2e62ab44af995165bd5116a9d43 [term 8 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260925 03:05:17.769740 27485 raft_consensus.cc:2726] T 420163a6cdce412ebc21c25fe3835638 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260925 03:05:17.787236 27603 raft_consensus.cc:2781] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Leader election won for term 9
I20260925 03:05:17.787990 27268 raft_consensus.cc:2726] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260925 03:05:17.789525 27671 raft_consensus.cc:2726] T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260925 03:05:17.818140 27053 raft_consensus.cc:2781] T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260925 03:05:17.864384 27690 log.cc:924] Time spent T 15861cd28e354598a3c6bfc4337c34bb P 1bc1f2e62ab44af995165bd5116a9d43: Append to log took a long time: real 0.182s user 0.000s sys 0.000s
I20260925 03:05:17.786486 27673 raft_consensus.cc:2726] T 0b5df303cd6a4d59aca5c18bbbff24ed P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260925 03:05:17.898490 24618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:05:17.899268 24618 raft_consensus.cc:3037] T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Advancing to term 10
I20260925 03:05:17.733673 27508 consensus_queue.cc:1035] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.054s
I20260925 03:05:17.810431 27667 raft_consensus.cc:504] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:17.900377 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:17.876214 27666 leader_election.cc:288] T 29637f5a4a504ce6ba29033ccd0cd716 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:17.877434 27589 raft_consensus.cc:504] T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:17.878043 27434 leader_election.cc:288] T 7e49248643df4131a1c42e74244e05c2 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 10 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:17.883296 27627 raft_consensus.cc:504] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:17.909801 27687 raft_consensus.cc:482] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:17.914291 27687 raft_consensus.cc:504] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:17.915222 24541 raft_consensus.cc:1226] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20260925 03:05:17.884268 27053 raft_consensus.cc:482] T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c [term 10 FOLLOWER]: Starting leader election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:05:17.915984 27053 raft_consensus.cc:3037] T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c [term 10 FOLLOWER]: Advancing to term 11
I20260925 03:05:17.803665 27515 raft_consensus.cc:504] T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:17.860553 27241 raft_consensus.cc:3037] T 367277daf8d04778b3fa4ed94d71580d P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:05:17.862406 27603 raft_consensus.cc:686] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c [term 9 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:05:17.897429 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:17.929085 24473 raft_consensus.cc:3037] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Advancing to term 10
I20260925 03:05:17.933305 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:17.934015 24469 raft_consensus.cc:2454] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 10.
I20260925 03:05:17.916450 27687 leader_election.cc:288] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:05:17.934983 24471 raft_consensus.cc:1226] T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 6. Current term is 7. Ops: []
I20260925 03:05:17.808081 24588 leader_election.cc:302] T 5278a60f97da4e10aecd50a8d9a9a43c P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
W20260925 03:05:17.942046 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 134066us
I20260925 03:05:17.942894 24621 raft_consensus.cc:2454] T 3d89396576884f7c8901dba54f783947 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 6.
I20260925 03:05:17.946702 24588 leader_election.cc:302] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:05:17.946873 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f1fac028a144058b92331c0d9b44ebf" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:17.947623 24613 raft_consensus.cc:2454] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 9.
I20260925 03:05:17.914885 27627 leader_election.cc:288] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 9 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:17.950574 24436 leader_election.cc:302] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:05:17.862078 27635 consensus_queue.cc:1035] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.077s
I20260925 03:05:17.965966 24467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f68804eb5a244d87bb395d50683c6012" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:17.969949 24467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "100c113a9fd84cc8adf2ec7b33ea0093" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:17.970651 24467 raft_consensus.cc:2454] T 100c113a9fd84cc8adf2ec7b33ea0093 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 4.
I20260925 03:05:17.950525 27385 raft_consensus.cc:2781] T 5278a60f97da4e10aecd50a8d9a9a43c P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260925 03:05:17.971899 27385 raft_consensus.cc:482] T 5278a60f97da4e10aecd50a8d9a9a43c P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:05:17.972275 27385 raft_consensus.cc:3037] T 5278a60f97da4e10aecd50a8d9a9a43c P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Advancing to term 8
I20260925 03:05:17.966517 27637 raft_consensus.cc:2781] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260925 03:05:17.977111 27630 leader_election.cc:288] T 100c113a9fd84cc8adf2ec7b33ea0093 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:05:17.963581 27488 raft_consensus.cc:2781] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260925 03:05:17.981035 27488 raft_consensus.cc:482] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:05:17.981361 27488 raft_consensus.cc:3037] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Advancing to term 10
I20260925 03:05:17.985949 27515 leader_election.cc:288] T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 9 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:05:17.951864 27040 raft_consensus.cc:504] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:17.987263 27589 leader_election.cc:288] T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:17.987834 27040 leader_election.cc:288] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 10 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:05:17.988014 27640 raft_consensus.cc:2726] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260925 03:05:17.989912 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48ca6ccd98194cf2af7b5278d15bf4ea" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:05:17.948333 27667 leader_election.cc:288] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:17.998553 24588 leader_election.cc:302] T 100c113a9fd84cc8adf2ec7b33ea0093 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:05:17.991592 24616 raft_consensus.cc:2454] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 10.
I20260925 03:05:17.999433 27460 raft_consensus.cc:504] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:18.003482 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b8e0d260a884ecdb20f79de46c36b19" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:18.004051 24466 raft_consensus.cc:3037] T 7b8e0d260a884ecdb20f79de46c36b19 P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Advancing to term 10
I20260925 03:05:18.004827 27460 leader_election.cc:288] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 9 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
W20260925 03:05:18.009994 27695 log.cc:924] Time spent T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.100s user 0.001s sys 0.000s
I20260925 03:05:18.012619 24618 raft_consensus.cc:2454] T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 10.
I20260925 03:05:18.012465 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be4e0aa984f4f2b81a58e382e434ad5" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:18.013967 24469 raft_consensus.cc:3037] T 0be4e0aa984f4f2b81a58e382e434ad5 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Advancing to term 9
I20260925 03:05:17.977054 27637 raft_consensus.cc:482] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:05:18.015673 24466 raft_consensus.cc:2454] T 7b8e0d260a884ecdb20f79de46c36b19 P 869b0a8575b54d08ae2817fad5dd42d5 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 10.
I20260925 03:05:17.938993 27384 raft_consensus.cc:504] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:18.018913 24588 leader_election.cc:302] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
W20260925 03:05:18.031101 24511 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 257553us
W20260925 03:05:18.032042 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 241778us
I20260925 03:05:18.032773 24511 leader_election.cc:302] T ac9357d144aa49789faaf31fe5e6bf89 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:17.952410 27622 raft_consensus.cc:504] T edcf6e6a422d4d198d4fbb0f7d3d845d P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:17.951172 27628 raft_consensus.cc:504] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:18.046214 24511 leader_election.cc:302] T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 1bc1f2e62ab44af995165bd5116a9d43, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:18.050948 24619 raft_consensus.cc:1226] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [term 10 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 9. Current term is 10. Ops: [9.6-9.6]
I20260925 03:05:18.021186 27241 raft_consensus.cc:2781] T 100c113a9fd84cc8adf2ec7b33ea0093 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260925 03:05:18.053799 24511 leader_election.cc:302] T 7993f9c3e2a44f408bbf5d1e395e2306 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:18.021848 27637 raft_consensus.cc:3037] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Advancing to term 10
I20260925 03:05:18.054723 24469 raft_consensus.cc:2454] T 0be4e0aa984f4f2b81a58e382e434ad5 P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 9.
I20260925 03:05:18.057117 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "100c113a9fd84cc8adf2ec7b33ea0093" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:18.061954 24540 raft_consensus.cc:2454] T 100c113a9fd84cc8adf2ec7b33ea0093 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 4.
W20260925 03:05:18.055790 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 309300us
I20260925 03:05:18.062933 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be4e0aa984f4f2b81a58e382e434ad5" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:05:18.064019 24514 leader_election.cc:302] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
W20260925 03:05:18.051612 27636 consensus_meta.cc:287] T 5278a60f97da4e10aecd50a8d9a9a43c P a04a37fc4aa7460cbfdbeeb4e451e1f5: Time spent flushing consensus metadata: real 0.606s user 0.005s sys 0.007s
I20260925 03:05:18.073750 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b030d84c670b48d5bff337ccc0c77626" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:18.074358 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f1fac028a144058b92331c0d9b44ebf" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:18.074834 24618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e725b82985746b39c0cc8f671c95353" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:05:18.047942 27627 raft_consensus.cc:2726] T ac9357d144aa49789faaf31fe5e6bf89 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260925 03:05:18.076277 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f68804eb5a244d87bb395d50683c6012" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:18.045406 27641 raft_consensus.cc:2781] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [term 10 FOLLOWER]: Leader election won for term 10
I20260925 03:05:18.078451 24543 raft_consensus.cc:2454] T b030d84c670b48d5bff337ccc0c77626 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 11.
I20260925 03:05:18.079259 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b8e0d260a884ecdb20f79de46c36b19" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:05:18.054154 27675 raft_consensus.cc:2726] T 04e068a0f80941dc9138cee063f53bab P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260925 03:05:18.023602 27603 consensus_queue.cc:237] T f22ddd7d1c444dc58f45a1fb0b3e02bc P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:18.100389 24620 raft_consensus.cc:1261] T 74443a0a1dc34d88a2f42e3b722db405 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260925 03:05:18.070880 27268 raft_consensus.cc:2726] T 7993f9c3e2a44f408bbf5d1e395e2306 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260925 03:05:18.047680 27040 consensus_queue.cc:1035] T e92f377a79354a6ebb4eec67303c13e3 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.040s
I20260925 03:05:18.089112 24539 raft_consensus.cc:2454] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 8.
I20260925 03:05:18.090975 24542 raft_consensus.cc:3037] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Advancing to term 10
I20260925 03:05:18.083344 27622 leader_election.cc:288] T edcf6e6a422d4d198d4fbb0f7d3d845d P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 7 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:05:18.091657 24547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48ca6ccd98194cf2af7b5278d15bf4ea" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:05:18.094542 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b627ecb4a5384ef392aa695e06599c39" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:05:18.106777 24617 raft_consensus.cc:3037] T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Advancing to term 9
I20260925 03:05:18.061959 27241 raft_consensus.cc:482] T 100c113a9fd84cc8adf2ec7b33ea0093 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:05:18.097918 27627 raft_consensus.cc:2781] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260925 03:05:18.098282 24514 leader_election.cc:302] T 43366a0a2f9d4344bf4e37ffd290ef8a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:18.078593 24548 raft_consensus.cc:2454] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 9.
I20260925 03:05:18.071213 27634 raft_consensus.cc:482] T 11bb46bce3564841b7674e6a693bd2e6 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:18.123627 27627 raft_consensus.cc:482] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:18.126864 27634 raft_consensus.cc:504] T 11bb46bce3564841b7674e6a693bd2e6 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:18.054011 27515 consensus_queue.cc:1046] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.039s
I20260925 03:05:18.057915 27503 consensus_queue.cc:1035] T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.038s
I20260925 03:05:18.060307 27671 consensus_queue.cc:1035] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.040s
I20260925 03:05:18.088167 27636 raft_consensus.cc:2726] T 5278a60f97da4e10aecd50a8d9a9a43c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260925 03:05:18.122390 24617 raft_consensus.cc:2454] T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 9.
I20260925 03:05:18.125241 24514 leader_election.cc:302] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:18.144276 24511 leader_election.cc:302] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:05:18.152513 24545 raft_consensus.cc:2454] T 5278a60f97da4e10aecd50a8d9a9a43c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 7.
I20260925 03:05:18.123482 27241 raft_consensus.cc:3037] T 100c113a9fd84cc8adf2ec7b33ea0093 P eeab93abfdb141a895002ec9f312296c [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:05:18.126880 27627 raft_consensus.cc:3037] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Advancing to term 10
I20260925 03:05:18.154428 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bb46bce3564841b7674e6a693bd2e6" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:18.159548 24620 raft_consensus.cc:2454] T 11bb46bce3564841b7674e6a693bd2e6 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 6.
I20260925 03:05:18.129120 27634 leader_election.cc:288] T 11bb46bce3564841b7674e6a693bd2e6 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:18.161042 24511 leader_election.cc:302] T 11bb46bce3564841b7674e6a693bd2e6 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:05:18.138743 27675 raft_consensus.cc:2726] T 43366a0a2f9d4344bf4e37ffd290ef8a P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260925 03:05:18.176792 27707 raft_consensus.cc:482] T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:18.178135 27707 raft_consensus.cc:504] T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:18.177435 27641 raft_consensus.cc:686] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [term 10 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:05:18.179430 27641 consensus_queue.cc:237] T 7b8e0d260a884ecdb20f79de46c36b19 P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:18.156533 27636 raft_consensus.cc:2726] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260925 03:05:18.155258 27647 raft_consensus.cc:482] T 367277daf8d04778b3fa4ed94d71580d P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:18.130901 27268 consensus_queue.cc:1035] T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.030s
I20260925 03:05:18.128640 27385 raft_consensus.cc:504] T 5278a60f97da4e10aecd50a8d9a9a43c P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:05:18.214231 27711 log.cc:924] Time spent T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.058s user 0.000s sys 0.000s
I20260925 03:05:18.214489 27384 leader_election.cc:288] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 10 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:05:18.175374 27368 raft_consensus.cc:2781] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Leader election won for term 9
I20260925 03:05:18.175961 27543 raft_consensus.cc:482] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:05:18.210817 27634 raft_consensus.cc:2781] T 11bb46bce3564841b7674e6a693bd2e6 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260925 03:05:18.214102 27628 leader_election.cc:288] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 9 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:18.163740 27632 consensus_queue.cc:1035] T b5f9b503b0ab43999d582015a367441e P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.065s
I20260925 03:05:18.255904 24542 raft_consensus.cc:2454] T 7b8e0d260a884ecdb20f79de46c36b19 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 10.
W20260925 03:05:18.261339 24511 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 100653us
I20260925 03:05:18.268993 24473 raft_consensus.cc:2454] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 10.
I20260925 03:05:18.175554 27699 consensus_queue.cc:1035] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.096s
I20260925 03:05:18.251925 27634 raft_consensus.cc:482] T 11bb46bce3564841b7674e6a693bd2e6 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:18.254204 27368 raft_consensus.cc:686] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 LEADER]: Becoming Leader. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:05:18.290688 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:18.291258 24548 raft_consensus.cc:2454] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 10.
W20260925 03:05:18.285991 27686 consensus_meta.cc:287] T 1414dcfe53f64a78ae413531eb8ec117 P eeab93abfdb141a895002ec9f312296c: Time spent flushing consensus metadata: real 0.507s user 0.010s sys 0.000s
I20260925 03:05:18.255539 27543 raft_consensus.cc:504] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:18.286837 27634 raft_consensus.cc:3037] T 11bb46bce3564841b7674e6a693bd2e6 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:05:18.297502 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e49248643df4131a1c42e74244e05c2" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:05:18.228798 27647 raft_consensus.cc:504] T 367277daf8d04778b3fa4ed94d71580d P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:18.264249 27053 raft_consensus.cc:504] T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:05:18.308460 27709 log.cc:924] Time spent T 06f4a7eaf6534e4986c9fd35911038e4 P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.184s user 0.001s sys 0.000s
I20260925 03:05:18.316146 24618 raft_consensus.cc:1226] T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c [term 11 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 10. Current term is 11. Ops: []
I20260925 03:05:18.298216 27721 raft_consensus.cc:482] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 1bc1f2e62ab44af995165bd5116a9d43)
W20260925 03:05:18.318517 27704 log.cc:924] Time spent T 0be4e0aa984f4f2b81a58e382e434ad5 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.086s user 0.001s sys 0.000s
I20260925 03:05:18.318822 27707 leader_election.cc:288] T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
W20260925 03:05:18.320837 27716 log.cc:924] Time spent T b030d84c670b48d5bff337ccc0c77626 P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.056s user 0.000s sys 0.001s
W20260925 03:05:18.321117 27710 log.cc:924] Time spent T e92f377a79354a6ebb4eec67303c13e3 P 1bc1f2e62ab44af995165bd5116a9d43: Append to log took a long time: real 0.093s user 0.000s sys 0.001s
I20260925 03:05:18.296058 27686 raft_consensus.cc:504] T 1414dcfe53f64a78ae413531eb8ec117 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:18.315764 27368 consensus_queue.cc:237] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:05:18.333096 27708 log.cc:924] Time spent T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.124s user 0.001s sys 0.000s
I20260925 03:05:18.328744 27385 leader_election.cc:288] T 5278a60f97da4e10aecd50a8d9a9a43c P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 8 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:05:18.341503 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcf6e6a422d4d198d4fbb0f7d3d845d" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:05:18.342134 24544 raft_consensus.cc:3037] T edcf6e6a422d4d198d4fbb0f7d3d845d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:05:18.345552 27053 leader_election.cc:288] T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 11 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:05:18.330860 27543 leader_election.cc:288] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:18.350147 27686 leader_election.cc:288] T 1414dcfe53f64a78ae413531eb8ec117 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 6 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:05:18.328325 27488 raft_consensus.cc:504] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:18.352775 27488 leader_election.cc:288] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 10 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:05:18.361721 27669 raft_consensus.cc:3037] T 442d4a29b1d743599f4b6b4871e26668 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Advancing to term 9
I20260925 03:05:18.355587 27721 raft_consensus.cc:504] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:18.366425 27637 raft_consensus.cc:504] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:18.366997 27721 leader_election.cc:288] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:05:18.367558 27328 raft_consensus.cc:3032] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [term 8 LEADER]: Stepping down as leader of term 8
I20260925 03:05:18.367947 27328 raft_consensus.cc:729] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [term 8 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:05:18.372803 27627 raft_consensus.cc:504] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:18.375231 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ac285175464fbda67211d9574db415" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:05:18.377877 24545 raft_consensus.cc:3037] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Advancing to term 10
W20260925 03:05:18.381932 27719 log.cc:924] Time spent T 6c81a1b3b616492ab6c964f45af36077 P eeab93abfdb141a895002ec9f312296c: Append to log took a long time: real 0.125s user 0.001s sys 0.000s
I20260925 03:05:18.383697 27647 leader_election.cc:288] T 367277daf8d04778b3fa4ed94d71580d P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:18.384018 24470 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55562 (request call id 1575) took 1106 ms. Trace:
I20260925 03:05:18.384392 26762 raft_consensus.cc:3032] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [term 9 LEADER]: Stepping down as leader of term 9
I20260925 03:05:18.386812 27241 raft_consensus.cc:504] T 100c113a9fd84cc8adf2ec7b33ea0093 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:18.384377 24470 rpcz_store.cc:276] 0925 03:05:17.277448 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 03:05:17.278619 (+ 1171us) service_pool.cc:227] Handling call
0925 03:05:18.383814 (+1105195us) spinlock_profiling.cc:243] Waited 496 ms on lock 0x7b54003f6954. stack: 00007f4373576033 00007f4373575f30 00007f4372ed4534 00007f437cbe08b4 00007f437cbe0849 00007f4376455471 00007f43764d42ae 00007f437b411c27 00007f4374af9c04 00007f4374af9b92 00007f4374af9ac1 00007f4374af9a3c 00007f4374af8cf2 00007f43748fa94c 00007f43748f9d86 00007f43748f972f
0925 03:05:18.383958 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1140310784}
I20260925 03:05:18.389361 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367277daf8d04778b3fa4ed94d71580d" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:18.390062 24613 raft_consensus.cc:2454] T 367277daf8d04778b3fa4ed94d71580d P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 7.
I20260925 03:05:18.390434 24544 raft_consensus.cc:2454] T edcf6e6a422d4d198d4fbb0f7d3d845d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 7.
I20260925 03:05:18.387457 26762 raft_consensus.cc:729] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [term 9 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:05:18.394007 24547 raft_consensus.cc:1226] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20260925 03:05:18.405316 24542 raft_consensus.cc:1261] T 293ef69caf514ac98510a68495f54eda P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Refusing update from remote peer 1bc1f2e62ab44af995165bd5116a9d43: Log matching property violated. Preceding OpId in replica: term: 5 index: 8. Preceding OpId from leader: term: 9 index: 10. (index mismatch)
I20260925 03:05:18.407326 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367277daf8d04778b3fa4ed94d71580d" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
W20260925 03:05:18.413009 24511 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 122785us
I20260925 03:05:18.409025 25163 leader_election.cc:302] T 29637f5a4a504ce6ba29033ccd0cd716 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:05:18.368482 27637 leader_election.cc:288] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 10 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:18.412029 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5278a60f97da4e10aecd50a8d9a9a43c" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:05:18.417418 24544 raft_consensus.cc:3037] T 5278a60f97da4e10aecd50a8d9a9a43c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Advancing to term 8
I20260925 03:05:18.417702 24547 raft_consensus.cc:1261] T ceab6b0fccd54e909a64f66fe81a9a1f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 10 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 10 index: 5. (index mismatch)
I20260925 03:05:18.422608 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c23337a7cb74184a0e57b7095cb6a11" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:18.424082 24614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35992316cb8d4c028333388eff02ecde" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:18.374650 27328 consensus_queue.cc:260] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:18.418946 27729 raft_consensus.cc:2781] T 29637f5a4a504ce6ba29033ccd0cd716 P 1bc1f2e62ab44af995165bd5116a9d43 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260925 03:05:18.439504 24614 raft_consensus.cc:2379] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 10: Already voted for candidate eeab93abfdb141a895002ec9f312296c in this term.
I20260925 03:05:18.422760 24545 raft_consensus.cc:2454] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 10.
I20260925 03:05:18.441067 24511 leader_election.cc:302] T 3d89396576884f7c8901dba54f783947 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:05:18.445309 27328 raft_consensus.cc:3037] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Advancing to term 9
I20260925 03:05:18.446384 24544 raft_consensus.cc:2454] T 5278a60f97da4e10aecd50a8d9a9a43c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 8.
I20260925 03:05:18.450089 27627 leader_election.cc:288] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 10 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:18.450898 27241 leader_election.cc:288] T 100c113a9fd84cc8adf2ec7b33ea0093 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:05:18.445472 24616 raft_consensus.cc:1226] T 7e49248643df4131a1c42e74244e05c2 P eeab93abfdb141a895002ec9f312296c [term 10 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 9. Current term is 10. Ops: []
I20260925 03:05:18.462154 27634 raft_consensus.cc:504] T 11bb46bce3564841b7674e6a693bd2e6 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:18.464954 27634 leader_election.cc:288] T 11bb46bce3564841b7674e6a693bd2e6 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 6 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:18.466477 26762 consensus_queue.cc:260] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:18.467609 26762 raft_consensus.cc:3037] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Advancing to term 10
I20260925 03:05:18.442241 27729 raft_consensus.cc:482] T 29637f5a4a504ce6ba29033ccd0cd716 P 1bc1f2e62ab44af995165bd5116a9d43 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:05:18.468603 27729 raft_consensus.cc:3037] T 29637f5a4a504ce6ba29033ccd0cd716 P 1bc1f2e62ab44af995165bd5116a9d43 [term 10 FOLLOWER]: Advancing to term 11
W20260925 03:05:18.468914 24436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 438681us
I20260925 03:05:18.473012 24588 leader_election.cc:302] T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:05:18.474668 27427 raft_consensus.cc:482] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:18.475194 27427 raft_consensus.cc:504] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:18.476063 26762 raft_consensus.cc:2781] T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Leader election won for term 9
I20260925 03:05:18.476428 24586 leader_election.cc:302] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5
W20260925 03:05:18.487826 24511 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 46821.3us
I20260925 03:05:18.489095 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcf6e6a422d4d198d4fbb0f7d3d845d" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:18.491708 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ac285175464fbda67211d9574db415" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:18.493918 24470 raft_consensus.cc:1261] T 293ef69caf514ac98510a68495f54eda P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Refusing update from remote peer 1bc1f2e62ab44af995165bd5116a9d43: Log matching property violated. Preceding OpId in replica: term: 7 index: 9. Preceding OpId from leader: term: 9 index: 10. (index mismatch)
W20260925 03:05:18.495051 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 433249us
I20260925 03:05:18.486721 25165 leader_election.cc:302] T 7e49248643df4131a1c42e74244e05c2 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:05:18.500795 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b5df303cd6a4d59aca5c18bbbff24ed" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:05:18.501327 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5332a5d329493991dff994e70ea92c" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:05:18.502525 24614 raft_consensus.cc:1261] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260925 03:05:18.503911 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b5df303cd6a4d59aca5c18bbbff24ed" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:05:18.504679 24547 raft_consensus.cc:1261] T f68804eb5a244d87bb395d50683c6012 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 8 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260925 03:05:18.505944 24540 raft_consensus.cc:1226] T edcf6e6a422d4d198d4fbb0f7d3d845d P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 6. Current term is 7. Ops: []
I20260925 03:05:18.506573 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f1fac028a144058b92331c0d9b44ebf" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:05:18.507186 24545 raft_consensus.cc:3037] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Advancing to term 10
I20260925 03:05:18.514220 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bb46bce3564841b7674e6a693bd2e6" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:05:18.517959 24464 raft_consensus.cc:1261] T ceab6b0fccd54e909a64f66fe81a9a1f P 869b0a8575b54d08ae2817fad5dd42d5 [term 10 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 10 index: 5. (index mismatch)
I20260925 03:05:18.520068 24544 raft_consensus.cc:1226] T b627ecb4a5384ef392aa695e06599c39 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 LEADER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 8. Current term is 9. Ops: []
I20260925 03:05:18.517904 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367277daf8d04778b3fa4ed94d71580d" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:18.521842 24473 raft_consensus.cc:2379] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1bc1f2e62ab44af995165bd5116a9d43 in current term 7: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:05:18.523962 27427 leader_election.cc:288] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
W20260925 03:05:18.528159 24468 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.511s user 0.012s sys 0.022s
W20260925 03:05:18.528601 24468 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.512s user 0.012s sys 0.022s
I20260925 03:05:18.533370 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5278a60f97da4e10aecd50a8d9a9a43c" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:18.534054 24464 raft_consensus.cc:3037] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Advancing to term 8
I20260925 03:05:18.536191 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e725b82985746b39c0cc8f671c95353" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:18.537657 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b627ecb4a5384ef392aa695e06599c39" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:18.538749 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35992316cb8d4c028333388eff02ecde" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:18.539803 24616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f1fac028a144058b92331c0d9b44ebf" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:05:18.541430 24616 raft_consensus.cc:3037] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Advancing to term 10
I20260925 03:05:18.492616 26762 raft_consensus.cc:686] T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c [term 9 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:05:18.542213 24472 raft_consensus.cc:2454] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 10.
I20260925 03:05:18.543901 24614 raft_consensus.cc:1226] T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 8. Current term is 9. Ops: []
I20260925 03:05:18.499102 27634 raft_consensus.cc:2781] T 3d89396576884f7c8901dba54f783947 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader election won for term 6
I20260925 03:05:18.545564 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bb46bce3564841b7674e6a693bd2e6" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:18.543596 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6b6ec010caf4689b8c90449764f5cf9" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:18.547439 24472 raft_consensus.cc:2379] T 11bb46bce3564841b7674e6a693bd2e6 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 6: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:05:18.552284 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:18.536252 24545 raft_consensus.cc:2454] T 9f1fac028a144058b92331c0d9b44ebf P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 10.
I20260925 03:05:18.553149 24466 raft_consensus.cc:2454] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 10.
I20260925 03:05:18.554332 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bb46bce3564841b7674e6a693bd2e6" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:18.555337 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5332a5d329493991dff994e70ea92c" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:18.556278 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9c33fd100ff44c1a3247319a324a044" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:18.556942 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35992316cb8d4c028333388eff02ecde" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:18.557631 24617 raft_consensus.cc:2454] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 10.
I20260925 03:05:18.561954 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c23337a7cb74184a0e57b7095cb6a11" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:18.562758 24473 raft_consensus.cc:2379] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in current term 10: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:05:18.563516 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7691c6b475428c82d6181d693d26f3" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:18.564000 24473 raft_consensus.cc:3037] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Advancing to term 10
W20260925 03:05:18.564872 25163 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 138033us
I20260925 03:05:18.566864 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1414dcfe53f64a78ae413531eb8ec117" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:18.567472 24471 raft_consensus.cc:3037] T 1414dcfe53f64a78ae413531eb8ec117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Advancing to term 6
I20260925 03:05:18.567617 25199 raft_consensus.cc:1226] T 7e49248643df4131a1c42e74244e05c2 P 1bc1f2e62ab44af995165bd5116a9d43 [term 10 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 9. Current term is 10. Ops: []
I20260925 03:05:18.568708 27609 consensus_queue.cc:1035] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20260925 03:05:18.569751 27729 raft_consensus.cc:504] T 29637f5a4a504ce6ba29033ccd0cd716 P 1bc1f2e62ab44af995165bd5116a9d43 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:18.570382 25197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6b6ec010caf4689b8c90449764f5cf9" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "1bc1f2e62ab44af995165bd5116a9d43" is_pre_election: true
I20260925 03:05:18.618618 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:05:18.553058 24464 raft_consensus.cc:2454] T 5278a60f97da4e10aecd50a8d9a9a43c P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 8.
I20260925 03:05:18.610134 24436 leader_election.cc:302] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:05:18.611634 24472 raft_consensus.cc:1226] T 7e49248643df4131a1c42e74244e05c2 P 869b0a8575b54d08ae2817fad5dd42d5 [term 10 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 9. Current term is 10. Ops: []
I20260925 03:05:18.577836 24465 raft_consensus.cc:1261] T 74443a0a1dc34d88a2f42e3b722db405 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Refusing update from remote peer a04a37fc4aa7460cbfdbeeb4e451e1f5: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
W20260925 03:05:18.619459 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 69995.3us
I20260925 03:05:18.633934 24545 raft_consensus.cc:3037] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 10 FOLLOWER]: Advancing to term 11
I20260925 03:05:18.634479 24437 leader_election.cc:302] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
W20260925 03:05:18.632710 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 112916us
I20260925 03:05:18.601722 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:05:18.627364 24466 raft_consensus.cc:3037] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [term 10 FOLLOWER]: Advancing to term 11
W20260925 03:05:18.573303 25165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 86715.3us
I20260925 03:05:18.637930 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "100c113a9fd84cc8adf2ec7b33ea0093" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:18.638531 24468 raft_consensus.cc:3037] T 100c113a9fd84cc8adf2ec7b33ea0093 P 869b0a8575b54d08ae2817fad5dd42d5 [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:05:18.600626 24616 raft_consensus.cc:2454] T 9f1fac028a144058b92331c0d9b44ebf P eeab93abfdb141a895002ec9f312296c [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 10.
I20260925 03:05:18.640322 24619 raft_consensus.cc:1226] T edcf6e6a422d4d198d4fbb0f7d3d845d P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 6. Current term is 7. Ops: []
I20260925 03:05:18.640398 25165 leader_election.cc:302] T 367277daf8d04778b3fa4ed94d71580d P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:05:18.641070 24621 raft_consensus.cc:3037] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [term 10 FOLLOWER]: Advancing to term 11
I20260925 03:05:18.646734 24545 raft_consensus.cc:2454] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 11.
I20260925 03:05:18.647964 24613 raft_consensus.cc:2454] T 11bb46bce3564841b7674e6a693bd2e6 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 6.
I20260925 03:05:18.650946 24467 raft_consensus.cc:1226] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [term 10 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 9. Current term is 10. Ops: []
I20260925 03:05:18.652122 24467 raft_consensus.cc:1226] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 8. Current term is 9. Ops: []
I20260925 03:05:18.542641 27536 raft_consensus.cc:482] T 442d4a29b1d743599f4b6b4871e26668 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
I20260925 03:05:18.655397 27536 raft_consensus.cc:504] T 442d4a29b1d743599f4b6b4871e26668 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:18.657368 27536 leader_election.cc:288] T 442d4a29b1d743599f4b6b4871e26668 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:05:18.654472 24621 raft_consensus.cc:2454] T 29637f5a4a504ce6ba29033ccd0cd716 P eeab93abfdb141a895002ec9f312296c [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 11.
I20260925 03:05:18.665272 24437 leader_election.cc:302] T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:18.665907 25162 leader_election.cc:302] T 29637f5a4a504ce6ba29033ccd0cd716 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:05:18.572623 27729 leader_election.cc:288] T 29637f5a4a504ce6ba29033ccd0cd716 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 11 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:18.673547 24465 raft_consensus.cc:1226] T 0be4e0aa984f4f2b81a58e382e434ad5 P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 8. Current term is 9. Ops: []
I20260925 03:05:18.667857 27543 raft_consensus.cc:482] T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:18.640862 27485 consensus_queue.cc:1046] T b627ecb4a5384ef392aa695e06599c39 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.053s
I20260925 03:05:18.693574 27543 raft_consensus.cc:504] T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:18.695799 27543 leader_election.cc:288] T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:18.697270 27707 raft_consensus.cc:482] T c85f5d1d3f004cf5817bb594fd8e6469 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:18.697854 27707 raft_consensus.cc:504] T c85f5d1d3f004cf5817bb594fd8e6469 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:18.573587 27722 raft_consensus.cc:2781] T 7e49248643df4131a1c42e74244e05c2 P 1bc1f2e62ab44af995165bd5116a9d43 [term 10 FOLLOWER]: Leader election won for term 10
I20260925 03:05:18.531518 27627 consensus_queue.cc:1046] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.106s
I20260925 03:05:18.576135 27605 raft_consensus.cc:2781] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260925 03:05:18.697180 24586 leader_election.cc:302] T edcf6e6a422d4d198d4fbb0f7d3d845d P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:05:18.673492 24466 raft_consensus.cc:2454] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 11.
I20260925 03:05:18.734608 24472 raft_consensus.cc:1226] T a9c33fd100ff44c1a3247319a324a044 P 869b0a8575b54d08ae2817fad5dd42d5 [term 11 FOLLOWER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 10. Current term is 11. Ops: [8.4-10.5]
I20260925 03:05:18.686843 24511 leader_election.cc:302] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43, a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:05:18.703857 24468 raft_consensus.cc:2454] T 100c113a9fd84cc8adf2ec7b33ea0093 P 869b0a8575b54d08ae2817fad5dd42d5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 5.
W20260925 03:05:18.715834 25165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 47212.6us
I20260925 03:05:18.617414 27634 raft_consensus.cc:686] T 3d89396576884f7c8901dba54f783947 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 LEADER]: Becoming Leader. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
W20260925 03:05:18.729372 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 81079.3us
W20260925 03:05:18.729662 25162 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 63851.7us
W20260925 03:05:18.663898 27670 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.972s user 0.045s sys 0.093s
W20260925 03:05:18.677263 27737 log.cc:924] Time spent T b030d84c670b48d5bff337ccc0c77626 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.081s user 0.000s sys 0.000s
I20260925 03:05:18.714691 27726 raft_consensus.cc:3037] T 367277daf8d04778b3fa4ed94d71580d P 1bc1f2e62ab44af995165bd5116a9d43 [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:05:18.618609 27589 consensus_queue.cc:1035] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.111s
I20260925 03:05:18.699191 27513 raft_consensus.cc:2781] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260925 03:05:18.752416 27513 raft_consensus.cc:482] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:05:18.715564 27667 raft_consensus.cc:2726] T 48ca6ccd98194cf2af7b5278d15bf4ea P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260925 03:05:18.679351 27508 raft_consensus.cc:2781] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260925 03:05:18.753140 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 81411.4us
I20260925 03:05:18.745749 27254 raft_consensus.cc:2781] T 29637f5a4a504ce6ba29033ccd0cd716 P 1bc1f2e62ab44af995165bd5116a9d43 [term 11 FOLLOWER]: Leader election won for term 11
I20260925 03:05:18.733618 27605 raft_consensus.cc:482] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:05:18.662132 27647 consensus_queue.cc:1035] T 293ef69caf514ac98510a68495f54eda P 1bc1f2e62ab44af995165bd5116a9d43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.087s
I20260925 03:05:18.749082 27722 raft_consensus.cc:686] T 7e49248643df4131a1c42e74244e05c2 P 1bc1f2e62ab44af995165bd5116a9d43 [term 10 LEADER]: Becoming Leader. State: Replica: 1bc1f2e62ab44af995165bd5116a9d43, State: Running, Role: LEADER
I20260925 03:05:18.675979 24471 raft_consensus.cc:2454] T 1414dcfe53f64a78ae413531eb8ec117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 6.
W20260925 03:05:18.788757 24511 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.4:43979, user_credentials={real_user=slave}} blocked reactor thread for 101752us
I20260925 03:05:18.746834 27627 raft_consensus.cc:3032] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 LEADER]: Stepping down as leader of term 9
I20260925 03:05:18.752738 27707 leader_election.cc:288] T c85f5d1d3f004cf5817bb594fd8e6469 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:18.756361 27267 consensus_queue.cc:1035] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260925 03:05:18.758803 23300 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260925 03:05:18.740778 27670 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.049s user 0.048s sys 0.103s
I20260925 03:05:18.752678 27634 consensus_queue.cc:237] T 3d89396576884f7c8901dba54f783947 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:18.770941 24437 leader_election.cc:302] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5; no voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:18.742506 27328 consensus_queue.cc:1035] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.051s
I20260925 03:05:18.714483 27629 consensus_queue.cc:1046] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: INVALID_TERM, Last received: 10.5, Next index: 6, Last known committed idx: 3, Time since last communication: 0.066s
I20260925 03:05:18.752832 27513 raft_consensus.cc:3037] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [term 10 FOLLOWER]: Advancing to term 11
W20260925 03:05:18.830821 24436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.3:35105, user_credentials={real_user=slave}} blocked reactor thread for 220798us
I20260925 03:05:18.772998 24473 raft_consensus.cc:2454] T 1d7691c6b475428c82d6181d693d26f3 P 869b0a8575b54d08ae2817fad5dd42d5 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 10.
W20260925 03:05:18.833490 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 62659us
I20260925 03:05:18.783617 27721 raft_consensus.cc:2741] T 29637f5a4a504ce6ba29033ccd0cd716 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 11 FOLLOWER]: Leader pre-election decision vote started in defunct term 10: won
I20260925 03:05:18.842212 24437 leader_election.cc:302] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:05:18.780441 27508 raft_consensus.cc:482] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [term 11 FOLLOWER]: Starting leader election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:18.795993 27605 raft_consensus.cc:3037] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Advancing to term 9
I20260925 03:05:18.843618 24514 leader_election.cc:302] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:18.844600 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35992316cb8d4c028333388eff02ecde" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:18.845305 24548 raft_consensus.cc:2454] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 9.
I20260925 03:05:18.793980 27254 raft_consensus.cc:686] T 29637f5a4a504ce6ba29033ccd0cd716 P 1bc1f2e62ab44af995165bd5116a9d43 [term 11 LEADER]: Becoming Leader. State: Replica: 1bc1f2e62ab44af995165bd5116a9d43, State: Running, Role: LEADER
I20260925 03:05:18.846551 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e8af2641e124f9589c5ccfdac785f63" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:18.847386 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c85f5d1d3f004cf5817bb594fd8e6469" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:18.851029 24548 raft_consensus.cc:1226] T 04ac285175464fbda67211d9574db415 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 10 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 9. Current term is 10. Ops: [9.6-9.6]
I20260925 03:05:18.851547 24514 leader_election.cc:302] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: eeab93abfdb141a895002ec9f312296c
I20260925 03:05:18.797093 27627 raft_consensus.cc:729] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:05:18.850342 24473 raft_consensus.cc:2454] T 9e8af2641e124f9589c5ccfdac785f63 P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 9.
I20260925 03:05:18.846333 27670 consensus_queue.cc:1046] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.007s
I20260925 03:05:18.853178 27671 raft_consensus.cc:2726] T 1e725b82985746b39c0cc8f671c95353 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260925 03:05:18.792263 27722 consensus_queue.cc:237] T 7e49248643df4131a1c42e74244e05c2 P 1bc1f2e62ab44af995165bd5116a9d43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 10, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } attrs { replace: true } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:18.835675 27589 raft_consensus.cc:2726] T 0b5df303cd6a4d59aca5c18bbbff24ed P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260925 03:05:18.919765 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637f5a4a504ce6ba29033ccd0cd716" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:18.920454 24468 raft_consensus.cc:3037] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [term 10 FOLLOWER]: Advancing to term 11
I20260925 03:05:18.858636 24514 leader_election.cc:302] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
W20260925 03:05:18.921353 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 62799.1us
I20260925 03:05:18.922062 24514 leader_election.cc:302] T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:18.923452 24514 leader_election.cc:302] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters: 1bc1f2e62ab44af995165bd5116a9d43, 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:05:18.924363 27543 raft_consensus.cc:2726] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
W20260925 03:05:18.924600 27736 log.cc:924] Time spent T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.088s user 0.000s sys 0.000s
I20260925 03:05:18.925060 27543 raft_consensus.cc:3037] T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Advancing to term 10
I20260925 03:05:18.929543 27629 raft_consensus.cc:482] T 550118bc1a0641ce9803bdbf43f2069f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:18.930132 27629 raft_consensus.cc:504] T 550118bc1a0641ce9803bdbf43f2069f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:18.932413 27629 leader_election.cc:288] T 550118bc1a0641ce9803bdbf43f2069f P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:18.830885 27479 consensus_queue.cc:1046] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.027s
I20260925 03:05:18.749477 26762 consensus_queue.cc:237] T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:18.850261 27485 raft_consensus.cc:2781] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [term 10 FOLLOWER]: Leader election won for term 10
I20260925 03:05:18.791267 27625 consensus_queue.cc:1046] T edcf6e6a422d4d198d4fbb0f7d3d845d P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.055s
I20260925 03:05:18.846990 27508 raft_consensus.cc:3037] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [term 11 FOLLOWER]: Advancing to term 12
I20260925 03:05:18.863243 27368 raft_consensus.cc:2781] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260925 03:05:18.916042 27586 raft_consensus.cc:482] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:18.850487 27667 consensus_queue.cc:1046] T b627ecb4a5384ef392aa695e06599c39 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.088s
I20260925 03:05:18.938678 27543 raft_consensus.cc:2726] T 2c23337a7cb74184a0e57b7095cb6a11 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260925 03:05:18.939170 27727 raft_consensus.cc:2726] T e6b6ec010caf4689b8c90449764f5cf9 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260925 03:05:18.957450 24468 raft_consensus.cc:2454] T 29637f5a4a504ce6ba29033ccd0cd716 P 869b0a8575b54d08ae2817fad5dd42d5 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bc1f2e62ab44af995165bd5116a9d43 in term 11.
I20260925 03:05:18.945194 27368 raft_consensus.cc:482] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 869b0a8575b54d08ae2817fad5dd42d5)
W20260925 03:05:18.959877 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 262686us
W20260925 03:05:18.958988 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 218075us
I20260925 03:05:18.941711 27671 consensus_queue.cc:1035] T 74443a0a1dc34d88a2f42e3b722db405 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.014s
I20260925 03:05:18.943789 27590 consensus_queue.cc:1046] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.018s
I20260925 03:05:18.949491 27485 raft_consensus.cc:686] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [term 10 LEADER]: Becoming Leader. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:05:18.972805 27667 consensus_queue.cc:1225] T b627ecb4a5384ef392aa695e06599c39 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Peer a04a37fc4aa7460cbfdbeeb4e451e1f5 log is divergent from this leader: its last log entry 9.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260925 03:05:18.980104 27513 raft_consensus.cc:504] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:18.986496 24586 leader_election.cc:302] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters: 869b0a8575b54d08ae2817fad5dd42d5
I20260925 03:05:18.987803 24586 leader_election.cc:302] T 5278a60f97da4e10aecd50a8d9a9a43c P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04a37fc4aa7460cbfdbeeb4e451e1f5, eeab93abfdb141a895002ec9f312296c; no voters:
W20260925 03:05:18.981081 24585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.4:43979, user_credentials={real_user=slave}} blocked reactor thread for 244892us
I20260925 03:05:18.949002 27629 consensus_queue.cc:1046] T 0be4e0aa984f4f2b81a58e382e434ad5 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.023s
I20260925 03:05:18.970613 27368 raft_consensus.cc:3037] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Advancing to term 10
I20260925 03:05:18.943071 27414 consensus_queue.cc:1046] T 7e49248643df4131a1c42e74244e05c2 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.023s
I20260925 03:05:18.958523 27460 raft_consensus.cc:3032] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 10 LEADER]: Stepping down as leader of term 10
I20260925 03:05:18.947970 27586 raft_consensus.cc:504] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:18.995316 27262 raft_consensus.cc:2781] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [term 10 FOLLOWER]: Leader election won for term 10
I20260925 03:05:18.995561 27460 raft_consensus.cc:729] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: a04a37fc4aa7460cbfdbeeb4e451e1f5, State: Running, Role: LEADER
I20260925 03:05:18.995908 27488 raft_consensus.cc:2781] T 5278a60f97da4e10aecd50a8d9a9a43c P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Leader election won for term 8
I20260925 03:05:18.996431 27488 raft_consensus.cc:686] T 5278a60f97da4e10aecd50a8d9a9a43c P eeab93abfdb141a895002ec9f312296c [term 8 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:05:18.996469 27460 consensus_queue.cc:260] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.5, Last appended by leader: 5, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:18.997370 27488 consensus_queue.cc:237] T 5278a60f97da4e10aecd50a8d9a9a43c P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:18.953114 27637 consensus_queue.cc:1046] T edcf6e6a422d4d198d4fbb0f7d3d845d P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.033s
I20260925 03:05:18.999308 27241 consensus_queue.cc:1046] T 0b5df303cd6a4d59aca5c18bbbff24ed P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260925 03:05:18.999157 27485 consensus_queue.cc:237] T 9f1fac028a144058b92331c0d9b44ebf P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:19.001199 27686 raft_consensus.cc:3032] T 0b5df303cd6a4d59aca5c18bbbff24ed P eeab93abfdb141a895002ec9f312296c [term 8 LEADER]: Stepping down as leader of term 8
I20260925 03:05:19.001557 27686 raft_consensus.cc:729] T 0b5df303cd6a4d59aca5c18bbbff24ed P eeab93abfdb141a895002ec9f312296c [term 8 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:05:19.002319 27686 consensus_queue.cc:260] T 0b5df303cd6a4d59aca5c18bbbff24ed P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:19.003610 27686 raft_consensus.cc:3037] T 0b5df303cd6a4d59aca5c18bbbff24ed P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Advancing to term 9
I20260925 03:05:18.970121 27640 raft_consensus.cc:2781] T edcf6e6a422d4d198d4fbb0f7d3d845d P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Leader election won for term 7
I20260925 03:05:19.005041 27640 raft_consensus.cc:686] T edcf6e6a422d4d198d4fbb0f7d3d845d P eeab93abfdb141a895002ec9f312296c [term 7 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:05:19.005914 27640 consensus_queue.cc:237] T edcf6e6a422d4d198d4fbb0f7d3d845d P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:19.007735 24588 leader_election.cc:302] T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:05:19.009081 26741 raft_consensus.cc:2781] T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c [term 11 FOLLOWER]: Leader election won for term 11
I20260925 03:05:19.009604 26741 raft_consensus.cc:686] T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c [term 11 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:05:19.010416 26741 consensus_queue.cc:237] T a9c33fd100ff44c1a3247319a324a044 P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 10.5, Last appended by leader: 5, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:19.011085 27384 consensus_queue.cc:1035] T ceab6b0fccd54e909a64f66fe81a9a1f P eeab93abfdb141a895002ec9f312296c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260925 03:05:18.972043 27705 raft_consensus.cc:482] T c85f5d1d3f004cf5817bb594fd8e6469 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:18.997773 27460 raft_consensus.cc:3037] T a9c33fd100ff44c1a3247319a324a044 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 10 FOLLOWER]: Advancing to term 11
I20260925 03:05:19.018821 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1414dcfe53f64a78ae413531eb8ec117" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:05:19.025988 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "100c113a9fd84cc8adf2ec7b33ea0093" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:05:19.023310 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9c33fd100ff44c1a3247319a324a044" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:05:19.020200 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7691c6b475428c82d6181d693d26f3" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
I20260925 03:05:19.026957 24544 raft_consensus.cc:3037] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Advancing to term 10
I20260925 03:05:18.970760 27254 consensus_queue.cc:237] T 29637f5a4a504ce6ba29033ccd0cd716 P 1bc1f2e62ab44af995165bd5116a9d43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.7, Last appended by leader: 7, Current term: 11, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:19.028537 27586 leader_election.cc:288] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:19.029150 25197 raft_consensus.cc:1261] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43 [term 8 FOLLOWER]: Refusing update from remote peer 869b0a8575b54d08ae2817fad5dd42d5: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260925 03:05:19.029836 27513 leader_election.cc:288] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 11 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:19.031169 27605 raft_consensus.cc:504] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:19.034195 27262 raft_consensus.cc:686] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [term 10 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:05:19.026626 24540 raft_consensus.cc:3037] T 100c113a9fd84cc8adf2ec7b33ea0093 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 4 FOLLOWER]: Advancing to term 5
I20260925 03:05:19.008581 24588 leader_election.cc:302] T 100c113a9fd84cc8adf2ec7b33ea0093 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:05:19.026407 24545 raft_consensus.cc:3037] T 1414dcfe53f64a78ae413531eb8ec117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Advancing to term 6
W20260925 03:05:19.036904 24514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 94350.7us
I20260925 03:05:19.036298 27705 raft_consensus.cc:504] T c85f5d1d3f004cf5817bb594fd8e6469 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:19.040441 24547 raft_consensus.cc:1226] T 4e5332a5d329493991dff994e70ea92c P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 10 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 9. Current term is 10. Ops: []
I20260925 03:05:19.043756 24544 raft_consensus.cc:2454] T 1d7691c6b475428c82d6181d693d26f3 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 10.
I20260925 03:05:19.103996 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442d4a29b1d743599f4b6b4871e26668" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" is_pre_election: true
I20260925 03:05:19.106518 24539 raft_consensus.cc:2379] T 442d4a29b1d743599f4b6b4871e26668 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c in current term 9: Already voted for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in this term.
I20260925 03:05:19.112978 27622 raft_consensus.cc:979] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P eeab93abfdb141a895002ec9f312296c: : Instructing follower a04a37fc4aa7460cbfdbeeb4e451e1f5 to start an election
I20260925 03:05:19.103965 24514 leader_election.cc:302] T 9e8af2641e124f9589c5ccfdac785f63 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, a04a37fc4aa7460cbfdbeeb4e451e1f5; no voters:
I20260925 03:05:19.116847 24545 raft_consensus.cc:2454] T 1414dcfe53f64a78ae413531eb8ec117 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 6.
W20260925 03:05:19.124526 24436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.4:43979, user_credentials={real_user=slave}} blocked reactor thread for 202948us
I20260925 03:05:19.036916 27407 consensus_queue.cc:1046] T 48ca6ccd98194cf2af7b5278d15bf4ea P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.083s
W20260925 03:05:19.134529 24437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 211681us
I20260925 03:05:19.139977 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c81a1b3b616492ab6c964f45af36077" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 9 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:19.120265 27427 raft_consensus.cc:482] T 11bb46bce3564841b7674e6a693bd2e6 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:19.157724 27262 consensus_queue.cc:237] T 04ac285175464fbda67211d9574db415 P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:19.164510 27590 raft_consensus.cc:482] T 442d4a29b1d743599f4b6b4871e26668 P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:19.165104 27590 raft_consensus.cc:504] T 442d4a29b1d743599f4b6b4871e26668 P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:19.167125 27590 leader_election.cc:288] T 442d4a29b1d743599f4b6b4871e26668 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:19.180059 27628 raft_consensus.cc:3032] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 LEADER]: Stepping down as leader of term 9
I20260925 03:05:19.180516 27628 raft_consensus.cc:729] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:05:19.181310 27628 consensus_queue.cc:260] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:19.182492 27628 raft_consensus.cc:3037] T 04ac285175464fbda67211d9574db415 P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Advancing to term 10
I20260925 03:05:19.182245 27368 raft_consensus.cc:504] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:19.184301 27368 leader_election.cc:288] T 35992316cb8d4c028333388eff02ecde P a04a37fc4aa7460cbfdbeeb4e451e1f5 [CANDIDATE]: Term 10 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:19.184773 27670 raft_consensus.cc:482] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:19.185240 27670 raft_consensus.cc:504] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:19.187283 27670 leader_election.cc:288] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:19.189245 27705 leader_election.cc:288] T c85f5d1d3f004cf5817bb594fd8e6469 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:05:19.120376 27718 raft_consensus.cc:482] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:05:19.190115 27718 raft_consensus.cc:504] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:19.128844 27725 raft_consensus.cc:1067] T abd5b8dc1b7e49e3aa9af900d73ff0e7 P eeab93abfdb141a895002ec9f312296c [term 8 LEADER]: Signalling peer a04a37fc4aa7460cbfdbeeb4e451e1f5 to start an election
W20260925 03:05:19.210223 27736 log.cc:924] Time spent T e92f377a79354a6ebb4eec67303c13e3 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.073s user 0.000s sys 0.000s
I20260925 03:05:19.194434 27726 raft_consensus.cc:2726] T 367277daf8d04778b3fa4ed94d71580d P 1bc1f2e62ab44af995165bd5116a9d43 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260925 03:05:19.195048 27241 raft_consensus.cc:2781] T 100c113a9fd84cc8adf2ec7b33ea0093 P eeab93abfdb141a895002ec9f312296c [term 5 FOLLOWER]: Leader election won for term 5
I20260925 03:05:19.218245 27705 raft_consensus.cc:3032] T 48ca6ccd98194cf2af7b5278d15bf4ea P eeab93abfdb141a895002ec9f312296c [term 6 LEADER]: Stepping down as leader of term 6
W20260925 03:05:19.295653 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 287124us
I20260925 03:05:19.301501 24588 leader_election.cc:302] T 1414dcfe53f64a78ae413531eb8ec117 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:05:19.195685 27427 raft_consensus.cc:504] T 11bb46bce3564841b7674e6a693bd2e6 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:19.271909 27513 raft_consensus.cc:482] T 8e5819c15eac4544a57e5ee6c9d4f117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader a04a37fc4aa7460cbfdbeeb4e451e1f5)
I20260925 03:05:19.316169 24616 raft_consensus.cc:1226] T b627ecb4a5384ef392aa695e06599c39 P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 8. Current term is 9. Ops: [7.6-8.7]
W20260925 03:05:19.293814 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 177746us
I20260925 03:05:19.305066 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442d4a29b1d743599f4b6b4871e26668" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:19.384368 24619 raft_consensus.cc:2454] T 442d4a29b1d743599f4b6b4871e26668 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 8.
I20260925 03:05:19.304629 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35992316cb8d4c028333388eff02ecde" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c"
I20260925 03:05:19.305922 24621 raft_consensus.cc:1226] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Rejecting Update request from peer 869b0a8575b54d08ae2817fad5dd42d5 for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20260925 03:05:19.385728 27427 leader_election.cc:288] T 11bb46bce3564841b7674e6a693bd2e6 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:19.265278 27589 consensus_queue.cc:1035] T 6c81a1b3b616492ab6c964f45af36077 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.077s
I20260925 03:05:19.305539 24588 leader_election.cc:302] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:05:19.388072 24618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c81a1b3b616492ab6c964f45af36077" candidate_uuid: "1bc1f2e62ab44af995165bd5116a9d43" candidate_term: 9 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:19.387990 27703 raft_consensus.cc:2781] T 1414dcfe53f64a78ae413531eb8ec117 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Leader election won for term 6
I20260925 03:05:19.388645 24621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c23337a7cb74184a0e57b7095cb6a11" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:19.389695 24621 raft_consensus.cc:2454] T 2c23337a7cb74184a0e57b7095cb6a11 P eeab93abfdb141a895002ec9f312296c [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 10.
I20260925 03:05:19.385876 24613 raft_consensus.cc:3037] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [term 10 FOLLOWER]: Advancing to term 11
I20260925 03:05:19.294616 27705 raft_consensus.cc:729] T 48ca6ccd98194cf2af7b5278d15bf4ea P eeab93abfdb141a895002ec9f312296c [term 6 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:05:19.397713 24613 raft_consensus.cc:2454] T 35992316cb8d4c028333388eff02ecde P eeab93abfdb141a895002ec9f312296c [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 11.
I20260925 03:05:19.397964 27705 consensus_queue.cc:260] T 48ca6ccd98194cf2af7b5278d15bf4ea P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:19.407045 25162 leader_election.cc:302] T 6c81a1b3b616492ab6c964f45af36077 P 1bc1f2e62ab44af995165bd5116a9d43 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1bc1f2e62ab44af995165bd5116a9d43; no voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c
I20260925 03:05:19.294003 27241 raft_consensus.cc:686] T 100c113a9fd84cc8adf2ec7b33ea0093 P eeab93abfdb141a895002ec9f312296c [term 5 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:05:19.414147 24436 leader_election.cc:302] T 442d4a29b1d743599f4b6b4871e26668 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:05:19.415592 24436 leader_election.cc:302] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
W20260925 03:05:19.430730 27751 log.cc:924] Time spent T 293ef69caf514ac98510a68495f54eda P 869b0a8575b54d08ae2817fad5dd42d5: Append to log took a long time: real 0.165s user 0.000s sys 0.001s
I20260925 03:05:19.423749 27703 raft_consensus.cc:686] T 1414dcfe53f64a78ae413531eb8ec117 P eeab93abfdb141a895002ec9f312296c [term 6 LEADER]: Becoming Leader. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:05:19.432870 27703 consensus_queue.cc:237] T 1414dcfe53f64a78ae413531eb8ec117 P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:19.405962 27669 raft_consensus.cc:482] T f22ddd7d1c444dc58f45a1fb0b3e02bc P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:19.435825 27669 raft_consensus.cc:504] T f22ddd7d1c444dc58f45a1fb0b3e02bc P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:19.437997 27669 leader_election.cc:288] T f22ddd7d1c444dc58f45a1fb0b3e02bc P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:19.423959 27605 leader_election.cc:288] T f68804eb5a244d87bb395d50683c6012 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 9 election: Requested vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:05:19.439254 27628 raft_consensus.cc:2781] T 442d4a29b1d743599f4b6b4871e26668 P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260925 03:05:19.439795 27628 raft_consensus.cc:482] T 442d4a29b1d743599f4b6b4871e26668 P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:05:19.440269 27628 raft_consensus.cc:3037] T 442d4a29b1d743599f4b6b4871e26668 P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Advancing to term 10
I20260925 03:05:19.444998 27670 consensus_queue.cc:1046] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260925 03:05:19.446280 27670 raft_consensus.cc:2781] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260925 03:05:19.446702 27670 raft_consensus.cc:482] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:05:19.447067 27670 raft_consensus.cc:3037] T 2c23337a7cb74184a0e57b7095cb6a11 P 869b0a8575b54d08ae2817fad5dd42d5 [term 10 FOLLOWER]: Advancing to term 11
I20260925 03:05:19.449028 27667 raft_consensus.cc:3032] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 LEADER]: Stepping down as leader of term 8
I20260925 03:05:19.449533 27667 raft_consensus.cc:729] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:05:19.450557 27667 consensus_queue.cc:260] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
W20260925 03:05:19.451269 27753 log.cc:924] Time spent T 3d89396576884f7c8901dba54f783947 P a04a37fc4aa7460cbfdbeeb4e451e1f5: Append to log took a long time: real 0.138s user 0.000s sys 0.001s
I20260925 03:05:19.452167 27667 raft_consensus.cc:3037] T f68804eb5a244d87bb395d50683c6012 P 869b0a8575b54d08ae2817fad5dd42d5 [term 8 FOLLOWER]: Advancing to term 9
W20260925 03:05:19.456105 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.193.2:36389, user_credentials={real_user=slave}} blocked reactor thread for 45780.5us
W20260925 03:05:19.456804 24588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.193.1:42187, user_credentials={real_user=slave}} blocked reactor thread for 151388us
I20260925 03:05:19.458176 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bb46bce3564841b7674e6a693bd2e6" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:19.458863 24619 raft_consensus.cc:2454] T 11bb46bce3564841b7674e6a693bd2e6 P eeab93abfdb141a895002ec9f312296c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869b0a8575b54d08ae2817fad5dd42d5 in term 6.
I20260925 03:05:19.461618 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442d4a29b1d743599f4b6b4871e26668" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:19.462245 24436 leader_election.cc:302] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:05:19.464371 24540 raft_consensus.cc:2454] T 100c113a9fd84cc8adf2ec7b33ea0093 P a04a37fc4aa7460cbfdbeeb4e451e1f5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeab93abfdb141a895002ec9f312296c in term 5.
I20260925 03:05:19.405200 27427 raft_consensus.cc:482] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:05:19.471716 24467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e8af2641e124f9589c5ccfdac785f63" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:19.473857 24465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f68804eb5a244d87bb395d50683c6012" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5"
I20260925 03:05:19.473979 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c85f5d1d3f004cf5817bb594fd8e6469" candidate_uuid: "eeab93abfdb141a895002ec9f312296c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "869b0a8575b54d08ae2817fad5dd42d5" is_pre_election: true
I20260925 03:05:19.474354 24464 raft_consensus.cc:1261] T 0be4e0aa984f4f2b81a58e382e434ad5 P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260925 03:05:19.474843 27625 raft_consensus.cc:2781] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [term 11 FOLLOWER]: Leader election won for term 11
I20260925 03:05:19.475775 27241 consensus_queue.cc:237] T 100c113a9fd84cc8adf2ec7b33ea0093 P eeab93abfdb141a895002ec9f312296c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:19.477303 24471 raft_consensus.cc:1261] T 7b8e0d260a884ecdb20f79de46c36b19 P 869b0a8575b54d08ae2817fad5dd42d5 [term 10 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260925 03:05:19.478277 24436 leader_election.cc:302] T 11bb46bce3564841b7674e6a693bd2e6 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b0a8575b54d08ae2817fad5dd42d5, eeab93abfdb141a895002ec9f312296c; no voters:
I20260925 03:05:19.481988 24616 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1e725b82985746b39c0cc8f671c95353"
dest_uuid: "eeab93abfdb141a895002ec9f312296c"
mode: GRACEFUL
new_leader_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
from {username='slave'} at 127.0.0.1:44570
I20260925 03:05:19.488804 24616 raft_consensus.cc:595] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Received request to transfer leadership to TS a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:05:19.489198 24616 raft_consensus.cc:603] T 1e725b82985746b39c0cc8f671c95353 P eeab93abfdb141a895002ec9f312296c [term 9 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260925 03:05:19.405988 27513 raft_consensus.cc:504] T 8e5819c15eac4544a57e5ee6c9d4f117 P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:19.489362 24467 raft_consensus.cc:2379] T 9e8af2641e124f9589c5ccfdac785f63 P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eeab93abfdb141a895002ec9f312296c in current term 9: Already voted for candidate 869b0a8575b54d08ae2817fad5dd42d5 in this term.
I20260925 03:05:19.491624 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f22ddd7d1c444dc58f45a1fb0b3e02bc" candidate_uuid: "869b0a8575b54d08ae2817fad5dd42d5" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:19.406255 27508 raft_consensus.cc:504] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:19.466358 27718 leader_election.cc:288] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 869b0a8575b54d08ae2817fad5dd42d5 (127.22.193.1:42187), a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389)
I20260925 03:05:19.483155 27605 consensus_queue.cc:1046] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.012s
I20260925 03:05:19.469238 27622 raft_consensus.cc:2781] T 1d7691c6b475428c82d6181d693d26f3 P eeab93abfdb141a895002ec9f312296c [term 10 FOLLOWER]: Leader election won for term 10
I20260925 03:05:19.502012 24622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c85f5d1d3f004cf5817bb594fd8e6469" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:19.503038 24613 raft_consensus.cc:1226] T 0be4e0aa984f4f2b81a58e382e434ad5 P eeab93abfdb141a895002ec9f312296c [term 9 LEADER]: Rejecting Update request from peer a04a37fc4aa7460cbfdbeeb4e451e1f5 for earlier term 8. Current term is 9. Ops: []
I20260925 03:05:19.469484 27750 raft_consensus.cc:482] T 4e5332a5d329493991dff994e70ea92c P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader eeab93abfdb141a895002ec9f312296c)
I20260925 03:05:19.510012 27589 raft_consensus.cc:2781] T 11bb46bce3564841b7674e6a693bd2e6 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260925 03:05:19.510486 27589 raft_consensus.cc:482] T 11bb46bce3564841b7674e6a693bd2e6 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:05:19.510871 27589 raft_consensus.cc:3037] T 11bb46bce3564841b7674e6a693bd2e6 P 869b0a8575b54d08ae2817fad5dd42d5 [term 6 FOLLOWER]: Advancing to term 7
I20260925 03:05:19.514284 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e8af2641e124f9589c5ccfdac785f63" candidate_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "eeab93abfdb141a895002ec9f312296c" is_pre_election: true
I20260925 03:05:19.514941 24617 raft_consensus.cc:2454] T 9e8af2641e124f9589c5ccfdac785f63 P eeab93abfdb141a895002ec9f312296c [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04a37fc4aa7460cbfdbeeb4e451e1f5 in term 8.
I20260925 03:05:19.488384 27427 raft_consensus.cc:504] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } attrs { replace: true } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } } peers { permanent_uuid: "1bc1f2e62ab44af995165bd5116a9d43" member_type: VOTER last_known_addr { host: "127.22.193.4" port: 43979 } attrs { promote: false } }
I20260925 03:05:19.510495 27750 raft_consensus.cc:504] T 4e5332a5d329493991dff994e70ea92c P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:19.517499 27427 leader_election.cc:288] T 367277daf8d04778b3fa4ed94d71580d P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105), 1bc1f2e62ab44af995165bd5116a9d43 (127.22.193.4:43979)
I20260925 03:05:19.505879 27625 raft_consensus.cc:686] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [term 11 LEADER]: Becoming Leader. State: Replica: 869b0a8575b54d08ae2817fad5dd42d5, State: Running, Role: LEADER
I20260925 03:05:19.518502 27750 leader_election.cc:288] T 4e5332a5d329493991dff994e70ea92c P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:19.519230 24467 raft_consensus.cc:1261] T f22ddd7d1c444dc58f45a1fb0b3e02bc P 869b0a8575b54d08ae2817fad5dd42d5 [term 9 FOLLOWER]: Refusing update from remote peer eeab93abfdb141a895002ec9f312296c: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260925 03:05:19.518877 27625 consensus_queue.cc:237] T 35992316cb8d4c028333388eff02ecde P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:19.521698 24613 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "367277daf8d04778b3fa4ed94d71580d"
dest_uuid: "eeab93abfdb141a895002ec9f312296c"
mode: GRACEFUL
new_leader_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5"
from {username='slave'} at 127.0.0.1:44570
I20260925 03:05:19.521893 27513 leader_election.cc:288] T 8e5819c15eac4544a57e5ee6c9d4f117 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:19.522315 24613 raft_consensus.cc:595] T 367277daf8d04778b3fa4ed94d71580d P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Received request to transfer leadership to TS a04a37fc4aa7460cbfdbeeb4e451e1f5
I20260925 03:05:19.522702 24613 raft_consensus.cc:603] T 367277daf8d04778b3fa4ed94d71580d P eeab93abfdb141a895002ec9f312296c [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260925 03:05:19.522416 27589 raft_consensus.cc:504] T 11bb46bce3564841b7674e6a693bd2e6 P 869b0a8575b54d08ae2817fad5dd42d5 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 } }
I20260925 03:05:19.524362 27589 leader_election.cc:288] T 11bb46bce3564841b7674e6a693bd2e6 P 869b0a8575b54d08ae2817fad5dd42d5 [CANDIDATE]: Term 7 election: Requested vote from peers a04a37fc4aa7460cbfdbeeb4e451e1f5 (127.22.193.2:36389), eeab93abfdb141a895002ec9f312296c (127.22.193.3:35105)
I20260925 03:05:19.524962 24472 raft_consensus.cc:1226] T b030d84c670b48d5bff337ccc0c77626 P 869b0a8575b54d08ae2817fad5dd42d5 [term 12 FOLLOWER]: Rejecting Update request from peer eeab93abfdb141a895002ec9f312296c for earlier term 11. Current term is 12. Ops: []
I20260925 03:05:19.469736 27637 consensus_queue.cc:1046] T b627ecb4a5384ef392aa695e06599c39 P 869b0a8575b54d08ae2817fad5dd42d5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3" port: 35105 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.037s
I20260925 03:05:19.528488 27605 raft_consensus.cc:3032] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c [term 9 LEADER]: Stepping down as leader of term 9
I20260925 03:05:19.528976 27605 raft_consensus.cc:729] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c [term 9 LEADER]: Becoming Follower/Learner. State: Replica: eeab93abfdb141a895002ec9f312296c, State: Running, Role: LEADER
I20260925 03:05:19.529750 27605 consensus_queue.cc:260] T 4e5332a5d329493991dff994e70ea92c P eeab93abfdb141a895002ec9f312296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869b0a8575b54d08ae2817fad5dd42d5" member_type: VOTER last_known_addr { host: "127.22.193.1" port: 42187 } } peers { permanent_uuid: "a04a37fc4aa7460cbfdbeeb4e451e1f5" member_type: VOTER last_known_addr { host: "127.22.193.2" port: 36389 } } peers { permanent_uuid: "eeab93abfdb141a895002ec9f312296c" member_type: VOTER last_known_addr { host: "127.22.193.3"